Enable thread-safe Catch2 assertions in framework tests

[knoepfel]

Summary

Several test algorithm bodies currently use plain assert(...) instead of Catch2 assertions. These checks disappear under NDEBUG, and a failure aborts the process rather than producing a Catch2 failure with expression diagnostics and test-case attribution.

Catch2 3.13.0, already used by this repository, supports assertions from worker threads when built with CATCH_CONFIG_THREAD_SAFE_ASSERTIONS. This issue should enable that supported Catch2 capability throughout the repository rather than use Phlex resources to serialize assertion-bearing algorithms.

Why Not A catch2_resource

A single-token catch2_resource would serialize every algorithm body that performs an assertion. That changes the concurrency regime under test solely to accommodate the test framework, which can invalidate or weaken tests intended to exercise parallel execution. It also couples Phlex production-facing resource semantics to an implementation detail of the test framework.

Resources remain the appropriate mechanism when the algorithm itself requires a constrained resource, such as ROOT, a database connection pool, a GPU context, or an external simulator. They should not be used merely to protect Catch2 reporting.

A local mutex is not a better alternative. It would have to cover Catch2's complete assertion-reporting path, including successful assertions, and would reimplement a supported Catch2 facility with worse maintenance and diagnostics.

Proposed Change

1. Configure Catch2 itself with CATCH_CONFIG_THREAD_SAFE_ASSERTIONS, before FetchContent_MakeAvailable(Catch2 ...), so Catch2's compiled implementation and every test translation unit use the same configuration.
2. Replace Catch2-linked assert calls in framework algorithm bodies with non-fatal CHECK-family assertions.
3. Remove Catch2-driven serialization and stale comments that claim CHECK cannot run in parallel, while retaining any concurrency limit that the test needs for its own semantics.
4. Add concise test-contributor guidance: code that can execute on a framework worker thread must use non-fatal CHECK-family assertions, not REQUIRE-family assertions or FAIL.
5. Exercise the relevant parallel tests repeatedly, including under TSAN where feasible.

Thread-Safety Rules

With CATCH_CONFIG_THREAD_SAFE_ASSERTIONS enabled, Catch2 supports concurrent runtime assertion reporting, including the CHECK family used by the current framework-worker callback sites. This preserves their intended concurrency::unlimited behavior.

The configuration does not make every macro appropriate for a worker thread:

• Use CHECK, CHECK_FALSE, CHECK_THROWS, CHECK_THROWS_AS, CHECK_THROWS_WITH, CHECK_NOTHROW, and CHECK_THAT in worker-thread code.
• Do not use REQUIRE, REQUIRE_FALSE, or FAIL in worker-thread code. A failing fatal assertion throws outside Catch2's test-case exception boundary and can terminate the process.
• Do not invoke SECTION, test-case, generator, or benchmark macros from worker threads.

Current State

The repository currently fetches Catch2 v3.13.0 but does not enable CATCH_CONFIG_THREAD_SAFE_ASSERTIONS. It only configures Catch2 counter handling. As a result, existing CHECK invocations from concurrent TBB callbacks are not protected by Catch2's thread-safe assertion implementation.

The current graph/TBB callback sites use only CHECK; no REQUIRE, REQUIRE_FALSE, or FAIL was found in those paths. Examples that can run concurrently include:

• test/provider_test.cpp: CHECK in concurrency::unlimited observers.
• test/unfold_test.cpp: CHECK in unlimited observers.
• test/multiple_function_registration_test.cpp: CHECK in unlimited transform callbacks.
• test/filter_test.cpp: CHECK in destructors of graph-owned objects, potentially on framework worker threads.

Some callbacks currently use concurrency::serial solely to avoid Catch2 thread-safety concerns, for example test/output_products_test.cpp. Serial execution prevents simultaneous invocations of that node but does not guarantee execution on Catch2's main test thread, so it is not a complete substitute for Catch2's thread-safe configuration.

Scope

In scope:

• Enable CATCH_CONFIG_THREAD_SAFE_ASSERTIONS for Catch2 globally and consistently.
• Convert Catch2-linked framework algorithm assertions from plain assert to CHECK where appropriate, including the existing workaround in test/class_registration_test.cpp.
• Revisit serialization or comments introduced solely for Catch2 assertion safety, preserving constraints that are independently meaningful to each test.
• Document the worker-thread assertion rule.
• Validate parallel assertion use, including sanitizer coverage where practical.

Out of scope:

• Converting assert calls in plugin, benchmark, and FORM translation units that are not linked against Catch2. Doing so would require a separate decision about their test/reporting architecture.
• Using the resources API merely as a Catch2 serialization mechanism.

[pcanal]

Now that #21 has landed a resource API, the concurrency hazard can be handled declaratively

On the other hand, doesn't it affect the semantic of the test (eg. making those algorithm run serially instead of concurrently)? Wouldn't it be better to take a lock if and only if we are about to throw the Catch2 'exception'?

[knoepfel]

On the other hand, doesn't it affect the semantic of the test (eg. making those algorithm run serially instead of concurrently)? Wouldn't it be better to take a lock if and only if we are about to throw the Catch2 'exception'?

@pcanal, yes, it could affect the semantics of the test. And it turns out that Catch2's thread-safety capabilities have improved. I had GPT-5.6 Terra evaluate the thread-safety consequences of the phlex repository. See below.

---

Prompt
If I understand https://github.com/catchorg/Catch2/blob/devel/docs/thread-safety.md correctly, any invocations of CHECK is thread-safe. Which Catch2 macros do we use in this code base, and which ones are problematic for thread-safety?

CHECK is only thread-safe in Catch2 when thread-safe assertions are enabled. Catch2 3.9+ makes this an opt-in feature, and this repository uses Catch2 v3.13.0 but does not enable CATCH_CONFIG_THREAD_SAFE_ASSERTIONS:

• Catch2 version: CMakeLists.txt:15-22
• Current Catch configuration only disables __COUNTER__: CMakeLists.txt:93-95, test/CMakeLists.txt:5-8

So, as currently built, every Catch2 runtime assertion invocation is unsafe when concurrently invoked from user/TBB worker threads, including CHECK.

Macros in use

Macro	Count	Thread-safety classification
TEST_CASE	189	Not safe to invoke from spawned threads; registration/test-case machinery is outside the supported thread-safe surface. Existing uses are normal test definitions, so this is not a current worker-thread issue.
SECTION	88	Not thread-safe from spawned threads. Existing uses are normal main test control flow.
CHECK	511	Becomes safe for concurrent assertion reporting only with CATCH_CONFIG_THREAD_SAFE_ASSERTIONS. Non-fatal, so it is the correct assertion family for worker-thread checks once enabled.
CHECK_FALSE	23	Same as CHECK.
CHECK_NOTHROW	25	Runtime assertion macro; requires the opt-in configuration. Non-fatal on assertion failure.
CHECK_THROWS	9	Same.
CHECK_THROWS_AS	70	Same.
CHECK_THROWS_WITH	36	Same.
CHECK_THAT	11	Same.
REQUIRE	79	The assertion machinery can be made thread-safe, but a failing REQUIRE from a worker thread throws an exception that is not caught by Catch2's test-case boundary, terminating the process. Do not use it in worker-thread callbacks.
REQUIRE_FALSE	1	Same failure behavior as REQUIRE.
FAIL	1	Like REQUIRE, it aborts by throwing; unsafe semantically in a spawned/worker thread because it can terminate the process.

No repository invocations were found for WARN, INFO, CAPTURE, SUCCEED, FAIL_CHECK, SKIP, generators, benchmarks, BDD macros, or static assertion macros.

Current parallel-context uses

The good news is that all Catch2 invocations found in graph/TBB callback paths are CHECK; no REQUIRE, FAIL, or section/generator macro was found in those paths.

The callbacks that can be concurrently active include:

• test/provider_test.cpp:94 and :122: CHECK in concurrency::unlimited observers.
• test/unfold_test.cpp:70, :72, and :79: CHECK in unlimited observers.
• test/multiple_function_registration_test.cpp:24 and :30: CHECK in unlimited transform callbacks.
• test/filter_test.cpp:36, :55, and :76: CHECK in destructors of graph-owned objects, whose destruction may occur on framework worker threads.

Those CHECKs are presently problematic because the Catch2 thread-safety option is not enabled. With it enabled, they are supported by Catch2.

Some callbacks use the default concurrency::serial, which prevents simultaneous execution of that graph node, but it does not guarantee execution on Catch2's main test thread. Thus serial callbacks should not be treated as a workaround for the missing Catch2 thread-safety configuration.

test/output_products_test.cpp:89 currently says that CHECK cannot run in a parallel context and forces serial concurrency. That statement is accurate for this build, but would become obsolete after enabling Catch2's thread-safe assertions. test/class_registration_test.cpp:47 makes a related comment while deliberately using standard assert.

Bottom line

• The documentation interpretation is correct only after enabling CATCH_CONFIG_THREAD_SAFE_ASSERTIONS.
• After enabling it, use CHECK and its non-fatal variants in worker threads; REQUIRE/REQUIRE_FALSE and FAIL remain unsuitable if they may fail on a worker thread.
• Never use SECTION, test-case, generator, or benchmark macros from worker threads.
• This repository's existing worker-thread assertion sites are all in the suitable CHECK family, but the project configuration currently leaves them non-thread-safe.