Skip to content

ci(ios): stop noise updates of the doc site screenshots - #4446

Merged
mfazekas merged 1 commit into
mainfrom
doc-screenshots-stable
Nov 11, 2026
Merged

mfazekas merged 1 commit into
mainfrom
doc-screenshots-stable

Conversation

@mfazekas

@mfazekas mfazekas commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Description

The nightly doc screenshot job opens a maps-docs PR almost every night for changes nobody wants (maps-docs#32). The main source is LineLayer/ThirdPartyVectorSource, which differs by 0.60–0.66% between any two nightly captures, just over the 0.5% threshold. This PR makes three changes:

  1. update-doc-screenshots.js uses a channel tolerance of 32 (was 16) and a threshold of 0.25% (was 0.5%).
  2. The nightly downloads the doc-screenshots artifact of the latest successful nightly that has one, and passes it to the script. A screenshot that differs from the docs is only updated if it also matches last night's capture. If it doesn't, it's listed under "Unstable, not updated" in the PR summary, or only in the job log when nothing else changed. Without a previous capture, the script logs that and keeps the current behaviour.
  3. With E2E_MODE: doc-screenshots, pick-ios-simulator.sh is pinned to iPhone 17 on iOS 26.5, the device the nightly picks today. If the runner image lacks either one, it fails and lists what the image has. Other modes still pick the newest plain iPhone.

Validation against the 9 nightly artifacts from 10-26 to 11-09, using the script's metric:

  • Noise since 10-30 at tolerance 32: ThirdPartyVectorSource is 0.025% (0.66% at 16). The remaining one-offs are DrawPolyline 0.38% and SetLanguage 0.24%.
  • Smallest real change: GetCenter (maps-docs#25) is 0.48% at tolerance 32, so it still passes 0.25%.
  • Replaying the nightlies from the 10-29 docs: the current rules commit something every night, including the blank offline maps from 11-04 (66%/38%/10%) and a revert of them two days later. The new rules commit GetCenter, ShowRegionDidChange and SetLanguage once, a night later than before. The blank offline maps and the DrawPolyline one-off are held back as unstable.
  • I ran the script locally on the 11-09 capture against maps-docs main, once with the 11-07 capture as previous and once with none, plus 11-04 vs 10-30. I also ran the download loop locally against this repo's runs. It skips the 11-08 run, which succeeded without a capture, and picks 42549819429. I didn't run the workflow itself, since it pushes to maps-docs.

Why ThirdPartyVectorSource varies: on the same simulator (iPhone 17, iOS 26.5), 10 fresh launches with a cleared tile cache, 5 with a warm cache, and 3 captured after 8 s instead of 30 s are all pixel-identical (0.000%). So it isn't GPU blending, tile load order or capture timing. All the lines are the same colour, so draw order and lineSortKey can't change the result either. The CI diffs are short stretches of line with Δ mostly 17–31, which is what one more or one fewer overlapping 0.6-opacity stroke produces. That points to the Mapillary tile content changing between nights, not to the renderer. I tried opacity 1.0 against simulated tile churn and it didn't reduce the diff, so the example is unchanged.

Checklist

  • I've read CONTRIBUTING.md
  • I updated the doc/other generated code with running yarn generate in the root folder
  • I have tested the new feature on /example app.
    • In V11 mode/ios
    • In New Architecture mode/ios
    • In V11 mode/android
    • In New Architecture mode/android
  • I added/updated a sample - if a new feature was implemented (/example)

Screenshot OR Video

Component to reproduce the issue you're fixing

Local-only harness used for the ThirdPartyVectorSource captures (registered as the app root, captured with xcrun simctl io screenshot after fresh launches). A reproVariant.json file next to it sets the opacity, colour and an optional tile URL for a local proxy that drops or shifts features.

import React from 'react';
import Mapbox, { MapView, Camera, VectorSource, LineLayer } from '@rnmapbox/maps';
import variant from './reproVariant.json';
import env from '../../env.json';

Mapbox.setAccessToken(env.accessToken);

const tileUrlTemplates = [
  'https://tiles.mapillary.com/maps/vtp/mly1_public/2/{z}/{x}/{y}?access_token=MLY|4142433049200173|72206abe5035850d6743b23a49c41333'.replaceAll('|', '%7C'),
];

export default function Repro() {
  return (
    <MapView style={{ flex: 1 }}>
      <Camera defaultSettings={{ centerCoordinate: [-87.622088, 41.878781], zoomLevel: 10 }} />
      <VectorSource id="mapillary" tileUrlTemplates={variant.tileUrl ? [variant.tileUrl] : tileUrlTemplates}>
        <LineLayer
          id="mapillary-lines"
          sourceLayerID="sequence"
          style={{ lineCap: 'round', lineJoin: 'round', lineOpacity: variant.opacity, lineColor: variant.color, lineWidth: 2.0 }}
        />
      </VectorSource>
    </MapView>
  );
}

Compare doc screenshots with a channel tolerance of 32 and a 0.25% threshold,
only update a changed screenshot when last night's capture agrees, and pin the
doc screenshot simulator to iPhone 17 / iOS 26.5.
@mfazekas
mfazekas merged commit 8f29895 into main Nov 11, 2026
10 checks passed
@mfazekas
mfazekas deleted the doc-screenshots-stable branch November 11, 2026 14:43

This branch was previously deployed

1 inactive deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant