Add safe PHP multithreading with isolated worker VMs

[TrixNEW]

Summary

zphp already uses native threads internally for zphp serve, but normal PHP applications still have no general-purpose way to execute PHP code in parallel across multiple CPU cores.

It would be useful to expose a safe threading API based on:

1 native thread = 1 isolated PHP VM

The goal should not be to make a single VM concurrently accessible from multiple threads.

Instead, zphp should provide general-purpose primitives that any PHP library/framework/application can build threading systems on top of.

Proposed model

Main Thread
└── PHP VM #0
      |
      +--> Worker Thread #1 -> PHP VM #1
      +--> Worker Thread #2 -> PHP VM #2
      +--> Worker Thread #3 -> PHP VM #3

Each worker should:

• own its VM for its entire lifetime
• remain persistent between tasks
• execute one task at a time
• communicate through safe value transfer/message passing
• never directly access another VM's mutable PHP state

This avoids needing locks/atomics throughout normal PHP objects, arrays, reference counting, GC, etc.

Example API

Something like:

$pool = new ZPHP\Parallel\Pool(4);

$future = $pool->submit(
    'expensiveFunction',
    [$data]
);

// main application can continue doing work

$result = $future->await();

$pool->shutdown();

Useful primitives could include:

$future->isComplete();
$future->await();
$future->cancel();

$pool->submit(...);
$pool->trySubmit(...);
$pool->shutdown();

The exact API isn't important; the underlying runtime support is.

Persistent workers

Workers should not create a new VM/thread for every task.

ThreadPool
├── Thread + persistent VM
├── Thread + persistent VM
├── Thread + persistent VM
└── Thread + persistent VM

Worker initialization should also be supported so applications can bootstrap things such as autoloaders or reusable worker-local state once.

For example:

$pool = new Pool(
    workers: 4,
    bootstrap: __DIR__ . '/worker.php'
);

Value transfer

Ordinary PHP objects/pointers should not be shared directly between VMs.

Initially, transferable types could include:

• null
• bool
• int
• float
• string
• arrays containing transferable values

Values should be copied/serialized into memory owned by the destination VM.

Large binary workloads could later use a transferable buffer where ownership is moved instead of copied.

Futures and completion

Submitted tasks should return a future.

Exceptions thrown inside a worker should propagate through that future rather than killing the worker:

try {
    $result = $future->await();
} catch (Throwable $e) {
    // worker task failed
}

Applications with event loops should also be able to check or receive completed tasks without busy-waiting.

Fiber integration could eventually allow await() to suspend a Fiber rather than block the OS thread.

Channels

A general-purpose channel primitive would also be useful:

$channel->send($value);
$value = $channel->receive();

This would allow applications to build their own:

• worker pools
• actors
• async task systems
• background workers
• server architectures

without unsafe shared PHP memory.

Backpressure

Task/message queues should be bounded.

For example:

new Pool(
    workers: 4,
    queueCapacity: 1024
);

This prevents an application from submitting work faster than workers can consume it and exhausting memory.

Extensions

Extensions should have clear lifecycle boundaries for multithreaded execution:

module init
worker init
task execution
worker shutdown
module shutdown

Worker-local extension state should only be accessed from its owning thread.

Process-global mutable extension state would remain the extension author's responsibility to synchronize.

Safety requirements

The implementation should avoid:

• sharing a VM between threads
• global VM locks
• unbounded queues
• thread creation per task
• VM creation per task
• forced thread termination
• executing PHP while internal queue locks are held

It should safely handle:

• exceptions
• cancellation
• worker failures
• shutdown while busy
• queue saturation
• worker initialization failures
• repeated pool creation/destruction

There should be no unresolved futures, deadlocks, leaks, use-after-free, or allocator ownership violations.

Performance

CPU-heavy tasks should genuinely execute simultaneously on multiple cores.

Programs that do not use threading should ideally have effectively no additional synchronization overhead.

Benchmarks should cover:

1 / 2 / 4 / 8 workers
task submission latency
CPU-bound scaling
large value transfer
memory usage
shutdown under load

Possible implementation phases

Phase 1

• persistent native worker threads
• isolated VM per worker
• named/static callable execution
• transferable primitive values/arrays
• futures
• exception propagation
• bounded task queue
• worker bootstrap
• safe shutdown

Phase 2

• channels
• cancellation/timeouts
• completion queues
• Fiber integration
• worker recovery
• better compiled-code reuse

Phase 3

• zero-copy transferable buffers
• richer transferable types
• native extension task integration

Acceptance criteria

This:

$pool = new ZPHP\Parallel\Pool(4);

$futures = [];

for ($i = 0; $i < 100; ++$i) {
    $futures[] = $pool->submit('expensiveCalculation', [$i]);
}

foreach ($futures as $future) {
    echo $future->await(), PHP_EOL;
}

$pool->shutdown();

should:

• execute PHP across multiple CPU cores
• use isolated persistent VMs
• safely transfer arguments/results
• propagate exceptions
• use bounded queues
• shut down cleanly
• avoid global VM locking
• not negatively affect normal single-threaded PHP

zphp already has native threading and persistent worker VMs for server mode, so exposing similar infrastructure as a safe general-purpose PHP parallelism API could make zphp useful for much more than request-based workloads.

[TrixNEW]

@nvms "Every once in a while, a revolutionary product comes along that changes everything." - Steve Jobs, introducing the iPhone in 2007.
LET'S MAKE PHP GREAT.

[TrixNEW]

also u got a discord? if ur interested in joining our dc we're using this project on, it'll prolly be easier to talk about zphp than in issues... if not it's fine idrm

[nvms]

This is a great idea. Very thorough too -- thanks for that. As mentioned, this kinda aligns already with how zphp serve already works, where each OS thread gets its own VM, and we already copy values between VMs, so most of this machinery already exists.

Building our own API for this is also a good idea instead of trying to align with, for example, ext-parallel, which is dated and pretty limiting.

The shape you propose is good. I'll think on this a bit and get to it soon!

I don't have discord, issues has been working for me for now, but maybe if this grows then a more persistent chat makes sense!

[nvms]

I've closed #71, #75, and #76 and folded them into this issue, since shared compiled code, transferable buffers, and cancellation all get designed together with the worker pool. This is now on the roadmap after the work currently in progress.