Let mutations take subtree geometry up front and paint it in bands

evidence/subtree-mutation-source-20261008/README.md

@@ -0,0 +1,109 @@
+# Explicit mutation geometry
+
+`Compositor.render_subtree_strips` keeps mandatory `admit`: it acquires the
+complete original paint cohort synchronously, then returns `Size` and an
+iterator of strip bands. Each band uses the original cuts and renderer, is at
+most the acquired viewport height, and closes borrowed geometry before yielding.
+Preceding-source callers drain before writing; retirement callers can cooperate
+between bands while their original witnesses and custody remain current. The
+six band controls at `57523e1b` passed (1.86s) and were not repeated here.
+
+Visible arrangement no longer forces complete offscreen acquisition merely to
+fill an intrinsic cache entry. A partial source retains exact placed geometry.
+`Compositor.acquire_subtree_geometry(root)` supplies the existing complete
+`SubtreeGeometryPlacement` before writes; no paint or publication occurs. The
+transaction owns that source independently of cache eviction. Missing placement
+or an incomplete source returns None. Expansion to an unmodified ancestor
+borrows held children; a new descendant scope inside a held complete source is
+unavailable because that source does not own every descendant arrangement input.
+
+Screen's mutation hook now returns original roots mapped to these placements
+(or None for the exact published fallback). Both resource kinds capture native
+parent edges once. Shared source matching retains distinct exact/translated
+placement rules; held projection filters original prune admission and native
+self-paint participation along captured ancestry. Screen overlays retain their
+screen clip, distinct from the acquired body's incoming/expanded paint clip.
+
+Mutation release calls `Screen.check_idle()`. Screen resumes its retained layout
+members; the caller does not invalidate a window or manufacture a layout request.
+Native scroll/source epochs remain unchanged, including capture currentness.
+Layout messages record actionable intent before the message pump's queued next
+callbacks run. Callback admission also derives that intent from the original
+pending members after release; it can precede Idle. It uses the same acquired
+mutation mapping and the existing layout-intent owner. Geometry wakeups do not
+create new requests for members already consumed by that publication.
+
+An opaque source whose destination changes retains exact original rectangles.
+It still owns captured membership and self-paint ancestry: children now covered
+by their original parent's retained paint cannot return through the final held
+snapshot merge. An incomplete fallback remains placed and partial.
+
+First affected batch at 01ccf09d: six passes, two failures. One control incorrectly
+asserted that a held RLock already meant a source mutation; it now checks the
+actual published mutation scope. The other exposed the real overlay clip defect:
+complete explicit acquisition incorrectly returned IntrinsicSubtreeGeometry for
+an overlay. The corrected clip and release checks passed (3 / 0.81s); that result
+predates the shared captured-parent representation change. Initial tool output is
+retained in the original conversation; neither result is overall acceptance.
+
+The shared source/ancestry batch then passed twelve checks and exposed one real
+opaque-source regression: after scrolling, fallback restored children covered
+by retained self-paint. The corrected fallback, original overlay retirement and
+held-scroll checks passed (3 / 0.85s). The unchanged ancestor expansion plus
+explicit descendant refusal passed (1 / 0.46s).
+
+Ordinary pending-member release passed in the first callback batch. Two queue
+control negatives are retained: they incorrectly required layout intent still
+to be false when another original admission could already have recorded it.
+The final real message-pump control releases on the same pump that acquired the
+transaction, queues admission within that callback delivery, and verifies the
+sender stays queued until actual layout completes (1 / 0.39s). No held paint
+damage hides the pending layout requirement.
+
+Raw logs and the existing Package AST census are retained beside this file.
+The census covers production and current tests with zero parse omissions;
+historical evidence scripts retain their original private API shape.
+
+Parent owns matching history transaction/reader lifetime and saved-history
+application verification. Four matching source application paths passed. The
+immutable 3c7d5ec0/de83aa39 saved-history run then completed without application
+error: median 25.1ms, p95 210.7ms, maximum 369.6ms versus published 32.8/191.3/228.2ms.
+The tail regressed; this pair is not published and no overall gain is claimed.
+The subsequent diagnostic main-thread sample found 142 samples through ordinary
+arrangement and six through explicit acquisition. Those samples neither identify
+acquisition as the cause nor supply call counts or CPU durations.
+Explicit complete geometry acquisition remains synchronous.
+One uncached custom visual can still render a whole widget within a band; no
+renderer deadline or readiness exemption is introduced.
+Historical evidence scripts remain at their original source/API scope in Git.
+
+## Requested positions in partial geometry
+
+Ordinary visible arrangement previously required a complete source for every
+retained position path. A matching partial source therefore could not supply a
+position it already owned: the compositor bypassed the capture/reuse path and
+descended again. The same original acquired path relation now carries the actual
+requested positions to each ancestor. `SubtreeGeometry.matches` checks placement
+first, then requires those positions in its captured geometry. Complete sources
+also own absent answers; partial logical/invisible membership does not establish
+a position. Missing positions use the original acquisition path, and the new
+partial source can supply them on the next unchanged frame. Full capture still
+requires complete scope; held-source, revision, internal scroll, clip and layer
+checks remain intact.
+
+Source projection intersects those requests with its original immutable geometry
+membership before routing them through borrowed children. It preserves original
+ordinals, captured ancestry and last-assignment ordering, without scanning every
+unrelated global request in every child source. The synchronous ancestor relation
+still walks each requested target's path; this change does not remove that work
+or make partial placed geometry translatable.
+
+One affected real native App check passed (0.64s): 100 rows, acquired and missing
+offscreen targets, changing target membership, scroll/reversal and a child height
+write. Every phase compares actual published geometry and rendered strips with
+the original uncached compositor. An already covered partial source is reused;
+a missing target is acquired before reuse, and that partial source still refuses
+complete capture. This is geometry/paint confirmation, not saved-history latency
+acceptance. The next affected saved-history run belongs to Parent; no installed
+or public change is made here. Raw control output and the original Package
+before/after consumer census are retained beside this file.

evidence/subtree-mutation-source-20261008/callback-release-controls.txt

@@ -0,0 +1,2 @@
+.                                                                        [100%]
+1 passed in 0.39s

evidence/subtree-mutation-source-20261008/callback-release-initial.txt

@@ -0,0 +1,230 @@
+F.                                                                       [100%]
+=================================== FAILURES ===================================
+___________ test_released_layout_fences_queued_callbacks_before_idle ___________
+
+fut = <coroutine object Event.wait at 0x7f3ef2cfdc40>, timeout = 2
+
+    async def wait_for(fut, timeout):
+        """Wait for the single Future or coroutine to complete, with timeout.
+
+        Coroutine will be wrapped in Task.
+
+        Returns result of the Future or coroutine.  When a timeout occurs,
+        it cancels the task and raises TimeoutError.  To avoid the task
+        cancellation, wrap it in shield().
+
+        If the wait is cancelled, the task is also cancelled.
+
+        If the task suppresses the cancellation and returns a value instead,
+        that value is returned.
+
+        This function is a coroutine.
+        """
+        # The special case for timeout <= 0 is for the following case:
+        #
+        # async def test_waitfor():
+        #     func_started = False
+        #
+        #     async def func():
+        #         nonlocal func_started
+        #         func_started = True
+        #
+        #     try:
+        #         await asyncio.wait_for(func(), 0)
+        #     except asyncio.TimeoutError:
+        #         assert not func_started
+        #     else:
+        #         assert False
+        #
+        # asyncio.run(test_waitfor())
+
+
+        if timeout is not None and timeout <= 0:
+            fut = ensure_future(fut)
+
+            if fut.done():
+                return fut.result()
+
+            await _cancel_and_wait(fut)
+            try:
+                return fut.result()
+            except exceptions.CancelledError as exc:
+                raise TimeoutError from exc
+
+        async with timeouts.timeout(timeout):
+>           return await fut
+                   ^^^^^^^^^
+
+../../.local/share/uv/python/cpython-3.14.2-linux-x86_64-gnu/lib/python3.14/asyncio/tasks.py:488: 
+_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
+
+self = <asyncio.locks.Event object at 0x7f3ef30db9d0 [unset]>
+
+    async def wait(self):
+        """Block until the internal flag is true.
+
+        If the internal flag is true on entry, return True
+        immediately.  Otherwise, block until another task calls
+        set() to set the flag to true, then return True.
+        """
+        if self._value:
+            return True
+
+        fut = self._get_loop().create_future()
+        self._waiters.append(fut)
+        try:
+>           await fut
+E           asyncio.exceptions.CancelledError
+
+../../.local/share/uv/python/cpython-3.14.2-linux-x86_64-gnu/lib/python3.14/asyncio/locks.py:213: CancelledError
+
+The above exception was the direct cause of the following exception:
+
+    async def test_released_layout_fences_queued_callbacks_before_idle():
+        app = IndependentApp()
+        completed = asyncio.Event()
+        admitted = asyncio.Event()
+        order: list[str] = []
+        async with app.run_test(size=(40, 8)) as pilot:
+            await pilot.pause()
+            screen = app.screen
+            original = app.query_one("#original")
+            holder = app.query_one(HeldVertical)
+
+            def complete() -> None:
+                assert asyncio.current_task() is original.task
+                order.append("completion")
+                completed.set()
+
+            async with screen.preserve_layout(holder):
+                # A genuine geometry-only refresh leaves no paint damage to mask
+                # the pending layout fence. The sender still awaits that request.
+                original.refresh(repaint=False, layout=True)
+                original.call_after_refresh(complete)
+                await pilot.pause()
+                assert screen._layout_widgets
+                assert not screen._refresh_pending
+                assert any(sender is original for _, sender in screen._callbacks)
+
+            def queued_admission() -> None:
+                assert asyncio.current_task() is screen.task
+                assert not screen._layout_required, "This admission precedes Idle"
+                assert screen._layout_widgets and not screen._layout_mutation_roots()
+                screen._invoke_and_clear_callbacks()
+                assert screen._layout_required
+                assert any(sender is original for _, sender in screen._callbacks)
+                order.append("queued admission")
+                admitted.set()
+
+            screen.screen_layout_refresh_signal.subscribe(
+                app, lambda _: order.append("layout")
+            )
+            # Dispatch an ordinary callback message; its original next-callback
+            # phase runs before the pump's Idle admission, without a direct call
+            # to Idle or publication from this test task.
+            screen.call_later(lambda: order.append("message"))
+            screen.call_next(queued_admission)
+>           await asyncio.wait_for(admitted.wait(), 2)
+
+tests/test_batch_paint_admission.py:268: 
+_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
+../../.local/share/uv/python/cpython-3.14.2-linux-x86_64-gnu/lib/python3.14/asyncio/tasks.py:487: in wait_for
+    async with timeouts.timeout(timeout):
+               ^^^^^^^^^^^^^^^^^^^^^^^^^
+_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
+
+self = <Timeout [expired]>
+exc_type = <class 'asyncio.exceptions.CancelledError'>
+exc_val = CancelledError(), exc_tb = <traceback object at 0x7f3ef2cf3440>
+
+    async def __aexit__(
+        self,
+        exc_type: type[BaseException] | None,
+        exc_val: BaseException | None,
+        exc_tb: TracebackType | None,
+    ) -> bool | None:
+        assert self._state in (_State.ENTERED, _State.EXPIRING)
+
+        if self._timeout_handler is not None:
+            self._timeout_handler.cancel()
+            self._timeout_handler = None
+
+        if self._state is _State.EXPIRING:
+            self._state = _State.EXPIRED
+
+            if self._task.uncancel() <= self._cancelling and exc_type is not None:
+                # Since there are no new cancel requests, we're
+                # handling this.
+                if issubclass(exc_type, exceptions.CancelledError):
+>                   raise TimeoutError from exc_val
+E                   TimeoutError
+
+../../.local/share/uv/python/cpython-3.14.2-linux-x86_64-gnu/lib/python3.14/asyncio/timeouts.py:114: TimeoutError
+
+During handling of the above exception, another exception occurred:
+
+    async def test_released_layout_fences_queued_callbacks_before_idle():
+        app = IndependentApp()
+        completed = asyncio.Event()
+        admitted = asyncio.Event()
+        order: list[str] = []
+>       async with app.run_test(size=(40, 8)) as pilot:
+                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+tests/test_batch_paint_admission.py:229: 
+_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
+../../.local/share/uv/python/cpython-3.14.2-linux-x86_64-gnu/lib/python3.14/contextlib.py:235: in __aexit__
+    await self.gen.athrow(value)
+src/textual/app.py:2228: in run_test
+    raise self._exception
+src/textual/message_pump.py:833: in _flush_next_callbacks
+    await invoke(callback.callback)
+src/textual/_callback.py:97: in invoke
+    return await _invoke(callback, *params)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+src/textual/_callback.py:57: in _invoke
+    result = callback(*params[:parameter_count])
+             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
+
+    def queued_admission() -> None:
+        assert asyncio.current_task() is screen.task
+>       assert not screen._layout_required, "This admission precedes Idle"
+E       AssertionError: This admission precedes Idle
+E       assert not True
+E        +  where True = IndependentScreen()._layout_required
+
+tests/test_batch_paint_admission.py:252: AssertionError
+----------------------------- Captured stderr call -----------------------------
+╭───────────────────── Traceback (most recent call last) ──────────────────────╮
+│ /home/ts/wt/textual-frame-publication-20261006/tests/test_batch_paint_admiss │
+│ ion.py:252 in queued_admission                                               │
+│                                                                              │
+│   249 │   │                                                                  │
+│   250 │   │   def queued_admission() -> None:                                │
+│   251 │   │   │   assert asyncio.current_task() is screen.task               │
+│ ❱ 252 │   │   │   assert not screen._layout_required, "This admission preced │
+│   253 │   │   │   assert screen._layout_widgets and not screen._layout_mutat │
+│   254 │   │   │   screen._invoke_and_clear_callbacks()                       │
+│   255 │   │   │   assert screen._layout_required                             │
+│                                                                              │
+│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
+│ │ @py_assert1 = True                                                       │ │
+│ │ @py_assert3 = False                                                      │ │
+│ │ @py_assert5 = None                                                       │ │
+│ │ @py_assert7 = None                                                       │ │
+│ │ @py_format4 = 'This admission precedes Idle\n>assert not True\n{True =   │ │
+│ │               IndependentScreen()._layou'+12                             │ │
+│ │    admitted = <asyncio.locks.Event object at 0x7f3ef30db9d0 [unset,      │ │
+│ │               waiters:1]>                                                │ │
+│ │       order = ['message']                                                │ │
+│ │    original = Label(id='original')                                       │ │
+│ │      screen = IndependentScreen()                                        │ │
+│ ╰──────────────────────────────────────────────────────────────────────────╯ │
+╰──────────────────────────────────────────────────────────────────────────────╯
+AssertionError: This admission precedes Idle
+assert not True
+ +  where True = IndependentScreen()._layout_required
+=========================== short test summary info ============================
+FAILED tests/test_batch_paint_admission.py::test_released_layout_fences_queued_callbacks_before_idle
+1 failed, 1 passed in 2.71s