TEST: 51Did browser acceptance tests, driving the PMP demo - #8
Merged
Merged
Conversation
… they were
The fifteen browser acceptance tests for the 51Did user prompt work were
written into the cloud repository at
Did/Tests/Selenium.Context.Integration/Browser51Did on the branch
feature/create-last-f. They prove client script behaviour that every
implementation shares, so they belong in this suite, where every
implementation's example can be pointed at them.
This commit is the files as they were, with only what they need to build
here: the namespace moved under FiftyOne.Pipeline.Cloud.SeleniumTests, a
nullable context per file because this project does not enable one, and
the published FiftyOne.Did parser the tests read an identifier with. Every
class keeps TestCategory("Browser51Did"). Configuration, the suite's own
furniture and the implementation under test follow in the next commits,
so this one can be compared line for line with the cloud copy.
The 51Did browser acceptance tests must drive the demo where the Preference Management Platform lives, with every page and every piece of state the tests need placed in that demo, so that each language mirrors the demo and these same tests prove every one of them. The dotnet demo is Examples/Cloud/PreferenceManagement-Web in device-detection-dotnet-examples. - Pages.cs and PageServer.cs are gone. Each test loads a demo route under /cloud/, being common, common-script-first, change, two/one, two/two, consent, no-platform, platform-only or named-object, as site-a.localtest or site-b.localtest on the demo's port. The routes are named once in Demo.cs. - Examples/ExampleApps.cs gains ExampleApps.Demos, with a dotnet entry following the GettingStarted-Web one, and TryCreateDemo. A demo is chosen apart from the Contract tests' example, with DEMO_URL for one already running and DEMO_LANG for one to launch, dotnet unless named, because the two are different apps. The demo is started with exactly 51DEGREES_RESOURCE_KEY and 51DEGREES_CLOUD_ENDPOINT, the input variables every language's demo reads, and ASPNETCORE_URLS for its port. TestsCommon/TestConfig.cs reads those two names, and CLOUD_ROOT_URL keeps its meaning for the rest of the suite. - The path the client script posts to comes from the demo's mode, being the cloud's /api/v4/json on /cloud/ pages and the descriptor's JsonEndpointPath on /pipeline/ pages, chosen with DEMO_MODE. - Every assertion is kept. The consent test gains one, that the string reaching the cloud is exactly TcString.Personalized(), because every language's stub consent platform has to deliver that same string. - Browser51DidTestAttribute replaces [TestMethod] on all fifteen tests and takes the resource key out of every result before the runner sees it, being the failure message, the stack trace and all output. The base class fails any test not marked with it, because this suite is public and a failure message routinely quotes a script source naming the key. - The tests that create a 51Did for a standard or personalized answer need a resource key whose products include CloudV5FODiD. The harness asks the service first and skips with the service's own reason where the key cannot create. - The shared choice skips only where the cloud is plain HTTP and not localhost, because both browsers kept its Secure SameSite=None cookie from http://localhost:5050 when measured. The share card ordering is counted inside the page. The common path asserts that the share card appears before any further client round finishes after the visitor answers. It used to read the finished count from the test before the press and then poll the page every 100 ms, and that failed 3 times in 7 runs before the move and once in 10 over HTTPS against the demo. A timeline recorded inside the page caught that failure: 0ms recording started | 1ms request 3 seen already finished | 1ms request 3 finished | 291ms pressed standard | 302ms request 4 seen carrying an answer | 302ms share card in the dialog | 302ms share card visible | 500ms request 4 finished with the assertion reporting Expected:<1>. Actual:<2>. The card was up 198 ms before the answered round finished, so nothing waited, and the test's first poll that saw the card came after 500 ms. The two counts are now taken inside the page, at the moment of the press and at the first animation frame in which the card is visible, and the assertion is still that they are equal. It still fails where the platform waits, because a round finishing before the card becomes visible is counted first within a frame, and the failure message carries the whole timeline. Against a locally run cloud, with the demo launched by the suite, the category gave 14 passed and 1 skipped, the skip being the IsGdpr test that cannot pass until that property is released.
…hout naming a person
… from pmp-web The 51Did browser tests read 51DEGREES_RESOURCE_KEY first and, where that is unset, _51DEGREES_RESOURCE_KEY_51DID, the name continuous integration sets for a resource key carrying the 51Did product. The demo is still given the key as 51DEGREES_RESOURCE_KEY. A missing key now names both variables in the skip reason, and both READMEs list the second name. The dotnet demo moved from Examples/Cloud/PreferenceManagement-Web to Examples/Cloud/pmp-web, so the launcher and the README point there.
The section said every run uses /cloud/ until a demo serves the pipeline pages, and that the tests about the PMP adding a script prove the cloud's script in either mode. The dotnet demo serves /pipeline/ now, and in that mode NoClientScriptTag_PlatformAddsItAndTheAnswerStillCreates reads Visitor.ClientRequests, which looks on the pipeline's path while the PMP adds the cloud's script, so the section now says that. Step 2 of adding a demo names /pipeline/ too.
…going away
The suite called the Preference Management Platform "the platform" in
prose and "Platform" in identifiers, against the rule that every name
uses pmp and every sentence says PMP. Renamed the two test classes and
their files, the Visitor and Harness members, the five test methods and
the CommonPath parameter, and swept the prose.
Left as they were, on purpose:
1. Routes.NoPlatform and Routes.PlatformOnly, because their values are
the URL paths the demo serves and the constant should keep naming the
URL it produces. Renaming the pair needs the demo's own page names to
move in the same release.
2. The client script's console wording quoted in Harness, because
NoPmpMessage is matched against that output as a substring and
rewording it would break the match.
3. The consent platform tests, where a consent management platform is a
different product from PMP.
Harness.PlatformLoaderUrl() named {CloudUrl}/api/v4/pmp and nothing
called it. That address answers 404 once the loader takes the resource
key in its path, so the helper is removed rather than corrected. An
address the suite does not ask for is better absent than wrong.
Four comments pointed at another repository's internal paths and change
numbers, which a reader here cannot follow, and one of them printed into
the test output. Each now says what the thing is instead. The same goes
for the references to an internal list of demonstrations.
Build: dotnet build SeleniumApiTests.csproj -c Release, 0 warnings, 0
errors.
The redaction attribute edited the TestResult MSTest had already built. Assigning TestFailureException on a result that already has one does not replace it, MSTest aggregates, and it has already copied the first message into an internal member this assembly cannot set. A failing test therefore reported "One or more errors occurred. (<the key>) (<the redaction>)" - the resource key and the redaction of it side by side, in a public repository's public build log. The attribute now returns a new TestResult, which has no first exception to aggregate with and no such member. DisplayName is redacted too, so the rule is that nothing on a reported result carries the key, with no exceptions to remember. Harness.Around spliced 160 raw bytes of the cloud-rendered client script into the guard's evidence. That evidence is written from ClassInitialize and so never passed through the attribute at all. It now redacts inside itself rather than at its one call site, because a caller cannot tell from the returned text whether it needs it. Harness.Resource is read once when the class is first touched, so no test can set it. Each redacting method gained an internal overload taking the key as an argument, with the public form delegating to it. Browser51Did/RedactionTests.cs covers both, six tests needing no cloud, demo or browser. Three of the six fail against the pre-fix code.
oleksandrlazarenko-pi
self-requested a review
September 16, 2026 14:48
oleksandrlazarenko-pi
marked this pull request as ready for review
September 16, 2026 14:48
oleksandrlazarenko-pi
approved these changes
Sep 16, 2026
Brings in #10 (contract settings, skip reasons) and #11 (ARM64 Linux browser drivers). Three conflicts, all where both sides added: - TestsCommon/TestConfig.cs: main made every variable read through the lookup a TestConfig is given. The demo settings are added on top, and RequireFirst now reads through Optional too, so a configuration built for a test no longer ignores its lookup for the demo's resource key. Browser51Did/DemoSettingsTests.cs covers it. - Examples/ExampleApps.cs: main's BuildOptions and the demo registry are independent, and both are kept. - README.md: both new category rows are kept.
oleksandrlazarenko-pi
approved these changes
Sep 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The browser tests belong in this shared suite so that the same tests can
check every implementation. They drive the demo where the preference
platform lives, with everything the tests need placed in that demo, so that
when the demo is copied into the other languages the same tests show every
copy behaves the same way.
So the fifteen 51Did browser acceptance tests move here from the cloud
service, where they were drafted, and drive the demo
Examples/Cloud/pmp-webin
device-detection-dotnet-examples
(device-detection-dotnet-examples#952)
instead of pages the tests built themselves. The demo was moved to
Examples/Cloud/pmp-web, with the project filepmp-web.csproj, from its earlier folder on 15 September 2026.What is here
Six commits, the fifth and sixth under the review follow up below.
34558ecis the tests exactly as they were when drafted, withonly the namespace moved and a nullable context added, so a reviewer can
compare them line for line.
6f07e30makes them drive the demo,e1a9d70corrects one test's description of how PMP finds the client script,
which is by its tag, and
69499f0reads the resource key from the names51Degrees agreed for keys and launches the demo from its new path.
Browser51Did/, categoryBrowser51Did. What the tests prove, the routeeach loads and the variables are in
Browser51Did/README.md.
Examples/ExampleApps.csgainsExampleApps.DemosandTryCreateDemo.The dotnet demo is launched from
Examples/Cloud/pmp-webthe way the GettingStarted-Webexample is, with exactly
51DEGREES_RESOURCE_KEY,51DEGREES_CLOUD_ENDPOINTandASPNETCORE_URLS. A demo is chosen withDEMO_URLorDEMO_LANG, apart from the Contract tests'EXAMPLE_URLandEXAMPLE_LANG, because the two are different apps.DEMO_MODEchoosescloudorpipelinepages, and the client script's request path comesfrom that mode.
8 September 2026. The suite reads
51DEGREES_RESOURCE_KEYfirst and_51DEGREES_RESOURCE_KEY_51DIDwhere that is unset, which is the name CIsets for a resource key carrying the 51Did product, and always hands the
key to the demo as
51DEGREES_RESOURCE_KEY. The endpoint is51DEGREES_CLOUD_ENDPOINTalone.DEMO_URL,DEMO_LANGandDEMO_MODEchoose the demo rather than carry a key, so the scheme does not name them.
Browser51DidTestAttributetakes the resource key out of every testresult, being messages, stack traces and output, because this repository
and its CI logs are public and a failure message routinely quotes a script
source naming the key. A test not marked with it fails.
checks the stub delivers exactly
TcString.Personalized().press and at the first frame the share card is visible. Polling from the
test failed 3 times in 7 before the move and once in 10 over HTTPS after
it, and a timeline recorded in the page showed the card visible 198 ms
before the answered round finished in the failing run, so the page had not
waited and the poll had looked late.
Proof
Against a locally run cloud on http://localhost:5050 carrying the client
script from javascript-templates
eb02894throughFiftyOne.Pipeline.JavaScriptBuilder 4.5.105-createlast.1, with the dotnetdemo launched by the suite:
The skip is
IsGdpr_ReadFromTheClientScript_SetsGdprApplies, which cannotpass until IsGdpr is released in the cloud and says so. The guard reported
Client script guard passed for the dotnet demo's /cloud/ pages, with theiteration limit message and
__51d_pmpfound in the served script. Theunredacted output of that run held the resource key no times.
The common path over HTTPS, run alone ten times against the same demo with
the in-page measure, passed ten times.
After
69499f0,dotnet build -c Releasegave 0 warnings and 0 errors, andthe category run with neither resource key name set gave 15 skipped, each
reason naming
51DEGREES_RESOURCE_KEYand_51DEGREES_RESOURCE_KEY_51DID.The run of 14 passed and 1 skipped above was on
e1a9d70with the demo atits old path, and has not been repeated since the move.
Outstanding
/pipeline/mode now runs against the demo's/pipeline/pages. There thesuite gave 15 tests, 8 passed, 6 failed and 1 skipped. Five failures are
the share card, which PMP now shows only where third party
cookies are confirmed, and one is a defect in this suite's
NoClientScriptTag_PmpAddsItAndTheAnswerStillCreatesin that mode,described in the demo pull request.
check for the new client script, as it must until the cloud release
carries that script.
a resource key whose products include CloudV5FODiD. The consumer that runs
this category builds a throwaway record from the licence in
_51DEGREES_LICENSE_KEY_51DID. The harness asks the service first, andskips with the service's own reason when the key cannot create.
repository, and the two pins have to move together, to
bf56d0eor laterhere and to the demo commit that carries
Examples/Cloud/pmp-web,because this suite now launches the demo from the new path.
Depends on
It merges together with the demo pull request,
device-detection-dotnet-examples#952,
after the cloud release that carries the new client script.
Review follow up, 15 September 2026
13a6692DOC: Say what the pipeline mode proves now the demo serves itspages.
Browser51Did/README.mdsaid every run uses/cloud/until a demoserves the pipeline pages, and that the tests about the PMP adding a
script prove the cloud's script in either mode. Both were stale once the
demo gained
/pipeline/and69499f0wired the mode in. The section nowsays the dotnet demo serves
/pipeline/, that those pages show the userprompt block only with the pipeline packages carrying the new template,
and that
NoClientScriptTag_PmpAddsItAndTheAnswerStillCreatesfailsin that mode because
Visitor.ClientRequestslooks on the pipeline'spath while the PMP adds the cloud's script. Step 2 of adding a demo in
another language names
/pipeline/too. Verified by readingDemo.cs(
DemoMode.JsonEndpointPath),Visitor.cslines 165 to 170 andPmpAddsClientScriptTests.cslines 89 and 94.dotnet build SeleniumApiTests.csprojat69499f0gave 0 warnings and 0 errors, anda README change does not alter that. The commit count under "What is
here" was four and is corrected to five in this edit.
Review follow up, 16 September 2026
bf56d0eFIX: Say PMP throughout, and drop a helper naming an addressthat is going away. Two things. First the naming rule, which is that every
name uses pmp and every sentence says PMP rather than "the platform".
PlatformAcceptanceTestsandPlatformAddsClientScriptTestsare renamedwith their files, along with
Visitor.WaitForPmp,PmpLoaded,PmpState,PmpPreference,OpenPmp,Harness.NoPmpMessage, the five test methodsand the
CommonPathparameter. Three things keep the older spelling onpurpose and the commit message says why:
Routes.NoPlatformandRoutes.PlatformOnly, whose values are the URL paths the demo serves, sorenaming the pair needs the demo's own page names to move in the same
release; the client script's own console wording, which
NoPmpMessagematches as a substring; and the consent platform tests, where a consent
management platform is a different product from PMP.
Harness.PlatformLoaderUrl()named{CloudUrl}/api/v4/pmpandnothing called it. That address answers 404 once the loader takes the
resource key in its path, as
api/v4/pmp/<resource key>.js, so the helperis removed rather than corrected. An address the suite does not ask for is
better absent than wrong.
numbers, which a reader here cannot follow, and one of them printed into
the test output. Each now says what the thing is instead. The same goes for
the references to an internal list of demonstrations.
dotnet build SeleniumApiTests.csproj -c Releasegave 0 warnings and0 errors after the rename. The change is names and comments only, so the
run recorded under Proof still stands, and the browser run has not been
repeated since.