Release abandoned orchestrator tasks when an executor is retired

[wangbill (YunchuWang)]

Problem

TaskOrchestrationContext creates a TaskCompletionSource<string> for every activity, sub-orchestration, and timer, registers it in openTasks, and awaits tcs.Task. When an episode ends while the orchestrator is parked on one of those awaits, the TCS is deliberately abandoned in a pending state — that is inherent to the replay model.

When a debugger is attached, TaskAwaiter.OnCompletedInternal calls OutputWaitEtwEvents, which calls Task.AddToActiveTasks(task) because Task.s_asyncDebuggingEnabled is true. That inserts the task into the process-wide, strongly-referenced static dictionary Task.s_currentActiveTasks. The matching Task.RemoveFromActiveTasks only runs inside the awaiter continuation wrapper, i.e. only when the awaited task completes.

Abandoned tasks are therefore rooted forever, and each rooted Task pins its continuation → async state machine → TaskOrchestrationContext → history, inputs, and outputs. Only the AddToActiveTasks branch roots the task (the ETW branch does not), which is exactly why the leak is debugger-specific.

Fixes the root cause behind Azure/azure-functions-durable-extension#340 ("Memory leak when debugger attached"), which is filed on the extension repo but labeled dtfx/external because the defect lives here in DurableTask.Core.

The fix

TaskOrchestrationContext gains internal void ReleaseOpenTasks(), which cancels every open task so the abandoned awaiter continuations run and unregister themselves. It is idempotent, sets isReleased = true before cancelling, and snapshots openTasks.Values.ToList() before iterating, because resumed user code can mutate the dictionary (concretely, CreateTimer's cancellation-token callback calls openTasks.Remove). A new private ThrowIfReleased() guard runs at the top of ScheduleTaskInternal, CreateSubOrchestrationInstanceCore, and CreateTimer so orchestrator code that swallows the cancellation cannot schedule new work that would leak the same way.

TaskOrchestrationExecutor gains internal void Release(). It restores the orchestrator ambient environment (TaskOrchestrationSynchronizationContext + OrchestrationContext.IsOrchestratorThread) because cancelling resumes orchestrator code synchronously, calls context.ReleaseOpenTasks(), and observes this.result.Exception if faulted so nothing surfaces as an UnobservedTaskException.

TaskOrchestrationDispatcher owns executor lifetime via workItem.Cursor, so it is the only component that knows when an executor will never run again. A new ReleaseCursor helper is called from a new finally on the outer try of OnProcessWorkItemSessionAsync (covering both the legacy Session == null path and the session path — workItem.Cursor is only ever assigned in OnProcessWorkItemAsync, which both paths call), and it replaces the bare workItem.Cursor = null; at the continue-as-new site.

⚠️ Continuation teardown is gated on Debugger.IsAttached

This is the part reviewers should look at hardest, and it is a deliberate trade-off rather than a free win.

Cancelling the abandoned tasks is what makes the CLR drop its roots — the entry is only removed when the awaiter continuation runs. There is no supported way to remove a s_currentActiveTasks root without running the async state-machine continuation. Running that continuation means the orchestrator's catch and finally blocks execute. So an unconditional release would change user-visible behavior in production to fix a bug that only exists under a debugger.

ReleaseCursor therefore always clears workItem.Cursor, but only calls TaskOrchestrationExecutor.Release() when System.Diagnostics.Debugger.IsAttached:

static void ReleaseCursor(ref OrchestrationExecutionCursor? cursor)
{
    ReleaseCursor(ref cursor, runContinuationTeardown: Debugger.IsAttached);
}

internal static void ReleaseCursor(ref OrchestrationExecutionCursor? cursor, bool runContinuationTeardown)
{
    OrchestrationExecutionCursor? retiredCursor = cursor;
    cursor = null;

    if (!runContinuationTeardown)
    {
        return;
    }

    retiredCursor?.OrchestrationExecutor?.Release();
}

The gate lives in the dispatcher because the dispatcher owns executor lifetime, and it is lifted into an internal overload taking the flag so both branches are unit-tested without needing an attached debugger in CI. No public API, no private-CLR reflection.

To be candid about what this does and does not claim:

• No debugger attached (i.e. production): behavior is exactly what it was before this PR. The executor is dropped and collected; no orchestrator continuation runs. The only cost is one Debugger.IsAttached check per retired cursor.
• Debugger attached: this is not a zero-behavior-change fix. The abandoned await is resumed with an OperationCanceledException, so orchestrator catch/finally blocks run during teardown. That is unavoidable with the public Task API. The compromise is now confined to the dev/debug session where the leak actually occurs.
• Escape hatch: run without a debugger attached. There is no AppContext switch convention in this repository, so this PR deliberately does not invent a settings surface for a single debug-only branch.
• Upstream fix: dotnet/runtime#26565 proposes making the active-task table weak. If that lands, this teardown becomes unnecessary and can be deleted outright.

Retirement is reported as InvalidOperationException, not cancellation

ThrowIfReleased() throws InvalidOperationException. The first failure at the abandoned await necessarily remains an OperationCanceledException — that is simply what a cancelled TCS raises. But a retired executor is permanent, so the extremely common

catch (OperationCanceledException) { /* retry */ }

shape must not treat it as a transient cancellation and spin. With InvalidOperationException, such a loop terminates on the retry.

A catch (Exception) infinite loop remains fundamentally impossible to prevent with any managed exception type — but it is now restricted to debugger teardown, and the guard still stops any new work from being scheduled.

Evidence

Direct-executor harness, 2000 episodes × fan-out 100, debugger simulated by setting Task.s_asyncDebuggingEnabled:

	Retained after full blocking GC	Entries leaked into Task.s_currentActiveTasks
Before	400.76 MB	1,006,000 (~210 KB/episode)
After	0.01 MB	0

With no debugger, both before and after are 0 — confirming the debugger-specific mechanism.

End-to-end through the real TaskHubWorker/TaskOrchestrationDispatcher with LocalOrchestrationService, 200 instances × 3 activities:

• Before: 4,200 leaked orchestrator tasks — 600 each of TaskOrchestration'4.<Execute>, the user orchestrator's <RunTask>, ScheduleTask, both ScheduleTaskToWorker overloads, ScheduleTaskInternal, plus ~600 TCS tasks. Exactly 3 abandoned awaits per instance.
• After: 0.

(A residual ~1,200 Task.Delay promises appear identically in both runs; that is the emulator's own queue polling, unrelated to this change.)

Public API

No public API change. TaskOrchestrationExecutor does not implement IDisposable and exposes no new members; the cleanup hook is internal void Release(), and the dispatcher gate is internal static void ReleaseCursor(ref OrchestrationExecutionCursor?, bool). Verified by reflecting over the built DurableTask.Core.dll rather than by reading source:

== DurableTask.Core.TaskOrchestrationExecutor  IsPublic=True
   interfaces: []
   PUBLIC   Constructor .ctor  (×3)
   PUBLIC   Method Execute
   PUBLIC   Method ExecuteNewEvents
   PUBLIC   Property IsCompleted
   internal method Release

Executor lifetime is owned by the dispatcher, which is the only component that knows when an executor will never run again, so there is nothing a caller outside this assembly could do with a public hook except use it incorrectly.

Extended sessions are explicitly unaffected

Release happens only at permanent executor retirement, never between episodes. Cancelling at the end of every episode would break extended sessions: HandleTaskCompletedEvent calls info.Result.SetResult(...), which throws InvalidOperationException on an already-cancelled TCS. There is a regression test, ExtendedSession_OpenTasksSurviveBetweenEpisodes, proving that an open task still survives across episodes and is resumed by its result rather than by cancellation.

Alternatives rejected

• Cancel at the end of every episode inside ExecuteCore — breaks extended sessions, as above.
• A custom awaitable that avoids registration — only fixes the innermost await; user-code awaits (Task.WhenAll, awaiting the returned Task<T>) still register.
• Reflectively removing entries from s_currentActiveTasks — private API, and it cannot reach the intermediate async-method tasks.

Tests

New file Test/DurableTask.Core.Tests/TaskOrchestrationExecutorTests.cs, 8 tests:

Test	Guarantee
Release_ResumesAbandonedOrchestratorContinuations	Every abandoned await is resumed so it can unregister itself
Release_WithAsyncDebuggingEnabled_DoesNotLeakActiveTasks	The leak itself is gone
Release_DoesNotChangeTheDecisionsAlreadyProduced	Teardown adds/removes no orchestrator actions
Release_OrchestratorThatSwallowsCancellation_CannotScheduleMoreWork	First failure is OperationCanceledException; the retry faults with InvalidOperationException, so cancellation-specific retry loops terminate
Release_IsIdempotent	Repeated release does not resume continuations twice
ReleaseCursor_WithTeardownEnabled_ClearsCursorAndResumesContinuations	Debugger branch: cursor cleared and continuations resumed
ReleaseCursor_WithTeardownDisabled_ClearsCursorWithoutRunningContinuations	Production branch: cursor cleared, zero user continuations run
ExtendedSession_OpenTasksSurviveBetweenEpisodes	The safety boundary — open tasks survive between episodes

Note on Release_WithAsyncDebuggingEnabled_DoesNotLeakActiveTasks: it deliberately measures how Task.s_currentActiveTasks growth scales with fan-out (1 vs 25 abandoned awaits over 20 episodes) rather than asserting an absolute count. The dictionary is process-wide and the test host's own async plumbing pollutes it, and on .NET Framework an async method's task is not an AsyncStateMachineBox<T>, so entries cannot be attributed by type there. The test is self-validating: it first asserts that unreleased executors DO leak proportionally, so it can never silently stop detecting the regression.

Results (Debug — test projects only build in Debug, since SIGN_ASSEMBLY disables InternalsVisibleTo)

• Full solution build: 0 warnings, 0 errors
• TaskOrchestrationExecutorTests: 8/8 net8.0, 8/8 net48
• DurableTask.Core.Tests: 147/147 net8.0; net48 128 passed / 7 failed
• DurableTask.Emulator.Tests: 5/5 net8.0, 5/5 net48

The 7 net48 failures are all in ContinueAsNewTraceBehaviorTests and are pre-existing on main, verified by stashing this change and re-running the same test class at baseline — identical 7 failures. No new failures.

Known follow-up (not fixed here)

azure-functions-durable-extension has its own secondary TCS leaks of the same shape that this change does not cover, because an orchestrator parked purely on an external event has no DTFx open tasks:

• DurableOrchestrationContext.pendingExternalEvents (EventTaskCompletionSource<T>)
• TaskCommonShim.timeoutTaskCompletionSource

[Copilot]

Pull request overview

This PR addresses a debugger-specific memory leak in DurableTask.Core by ensuring that “abandoned” orchestrator awaits (open activity/sub-orchestration/timer tasks) are released when an orchestration executor is retired, allowing CLR async-debug bookkeeping (Task.s_currentActiveTasks) to unregister those tasks.

Changes:

• Add TaskOrchestrationContext.ReleaseOpenTasks() and a released-context guard to prevent scheduling new work after release.
• Make TaskOrchestrationExecutor implement IDisposable and release open tasks on disposal while restoring orchestrator ambient execution context.
• Ensure TaskOrchestrationDispatcher disposes retired executors via a centralized ReleaseCursor(...) helper.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
Test/DurableTask.Core.Tests/TaskOrchestrationExecutorTests.cs	Adds regression tests for executor retirement/leak behavior, but currently placed in the legacy Test/ tree (not in the active test project).
src/DurableTask.Core/TaskOrchestrationExecutor.cs	Implements IDisposable to release abandoned orchestrator continuations when the executor is retired.
src/DurableTask.Core/TaskOrchestrationDispatcher.cs	Disposes retired executors in finally at session end and when continuing-as-new, via ReleaseCursor.
src/DurableTask.Core/TaskOrchestrationContext.cs	Adds open-task cancellation (ReleaseOpenTasks) and prevents scheduling new open tasks after release (ThrowIfReleased).

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

Test/DurableTask.Core.Tests/TaskOrchestrationExecutorTests.cs:18

• This test file is being added under Test/ (capital T), but the solution references the active Core test project at test/DurableTask.Core.Tests/DurableTask.Core.Tests.csproj. As a result, this file will not be compiled or executed by the test project, so the new regression coverage won’t actually run in CI.

namespace DurableTask.Core.Tests
{
    using System;
    using System.Collections.Generic;
    using System.Linq;

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (1)

Test/DurableTask.Core.Tests/TaskOrchestrationExecutorTests.cs:18

• This new test file is under Test/ (capital T), but the active test project is test/DurableTask.Core.Tests/DurableTask.Core.Tests.csproj and only compiles sources under that directory. As a result, these tests won't run in CI, so the new Release/ReleaseCursor behavior effectively has no coverage. Please move this file into test/DurableTask.Core.Tests/ (or otherwise include it in the active test .csproj).

namespace DurableTask.Core.Tests
{
    using System;
    using System.Collections.Generic;
    using System.Linq;

[wangbill (YunchuWang)]

Closing in favor of documentation. The proposed cleanup must run abandoned orchestrator continuations to release CLR debugger task roots, which changes user-code behavior while debugging. The underlying CLR weak-reference fix remains tracked by dotnet/runtime#26565.