Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
129 changes: 127 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,11 @@ Pure non-blocking (fiber based) strictly typed full-featured PHP driver for NATS
- [Fetch Batch](#fetch-batch)
- [Fetch Bytes](#fetch-bytes)
- [Fetch Immediate](#fetch-immediate)
- [Next message](#next-message)
- [Acknowledgements](#acknowledgements)
- [Push Consumer](#push-consumer)
- [Pull Consumer](#pull-consumer)
- [Ordered Consumer](#ordered-consumer)
- [Get message](#get-message)
- [NATS KV](https://docs.nats.io/nats-concepts/jetstream/key-value-store)
- [Store key values](#store-key-values)
Expand Down Expand Up @@ -798,7 +801,7 @@ And you can also configure the timeout and heartbeat settings.

#### Fetch Immediate

If you simply want to retrieve a batch of messages currently available in the stream, use `FetchConfig::immediate()`:
If you simply want to retrieve a batch of messages currently available in the stream, use `FetchConfig::immediate()`, optionally with the maximum number of messages to take:

```php
<?php
Expand All @@ -812,14 +815,53 @@ $consumer = $stream->createOrUpdateConsumer(...);

$batch = $consumer
->pulling()
->fetch(JetStream\FetchConfig::immediate());
->fetch(JetStream\FetchConfig::immediate(100));

foreach ($batch as $delivery) {
// handle message
$delivery->ack();
}
```

#### Next message

`PullConsumer::next()` waits for a single message, at most `maxWait` (30 seconds by default), and returns `null` if none came in time:

```php
<?php

declare(strict_types=1);

use Thesis\Time\TimeSpan;

$delivery = $consumer->pulling()->next(TimeSpan::fromSeconds(5));

if ($delivery !== null) {
// handle message
$delivery->ack();
}
```

#### Acknowledgements

`Delivery::ack()` sends the acknowledgement without waiting for the server. `Delivery::ackSync()` waits until the server confirms it has processed the acknowledgement,
so the message is not redelivered afterwards, and throws if the server does not confirm it, e.g. the consumer was deleted meanwhile.
`nack()`, `terminate()` and `inProgress()` complete the set.

```php
<?php

declare(strict_types=1);

use Amp\TimeoutCancellation;

$delivery = $consumer->pulling()->next();

$delivery?->ackSync(new TimeoutCancellation(5));
```

A consumer created with `headersOnly: true` delivers the headers without the payload, `Delivery::payloadSize()` then tells the size of the stored payload (for other consumers it is the size of the payload delivered).

#### Push Consumer

NATS offers two message delivery models: **pull** and **push**. Although the NATS documentation [recommends](https://docs.nats.io/nats-concepts/jetstream/consumers#dispatch-type-pull-push) the pull approach, for most PHP use cases push consumers are often preferable.
Expand Down Expand Up @@ -897,6 +939,24 @@ $consumer = $stream->createOrUpdateConsumer(new Api\ConsumerConfig(

Please refer to the push consumer configuration [documentation](https://docs.nats.io/nats-concepts/jetstream/consumers#push-specific) to understand the purpose of each parameter.

With `idleHeartbeat` set and without a `deliverGroup`, the subscription compares the heartbeats with the last message received: if the server has delivered messages
the subscription never got, e.g. dropped beyond the pending limits, a `ConsumerSequenceMismatch` is reported to `Client::onError()` once per gap.
The subscription keeps going, and the exception tells the stream sequence to read the lost messages again from:

```php
<?php

declare(strict_types=1);

use Thesis\Nats;

$nc->onError(static function (\Throwable $e): void {
if ($e instanceof Nats\Exception\ConsumerSequenceMismatch) {
dump("Lost messages of {$e->consumer}, resume from {$e->streamResumeSequence}");
}
});
```

#### Pull Consumer

Additionally, you can use `PullConsumer::consume`. This method handles message buffering, periodically requests the server and monitors timeouts to automatically request the next message batch.
Expand Down Expand Up @@ -990,6 +1050,71 @@ $consumer->consume(
);
```

Messages can also be read from an iterator with `PullConsumer::consumeIterator()`: the next message is taken only once the previous one has been read.

```php
<?php

declare(strict_types=1);

use Thesis\Nats\JetStream;

$iterator = $consumer->pulling()->consumeIterator(new JetStream\PullConsumeConfig(maxMessages: 100));

foreach ($iterator as $delivery) {
dump($delivery->message->payload);
$delivery->ack();

if ($done) {
$iterator->stop();
}
}
```

#### Ordered Consumer

An ordered consumer reads a stream in order, without gaps and without acknowledgements, e.g. to replay or to inspect it. The client creates an ephemeral push consumer with flow control and heartbeats
and replaces it with a new one, starting right after the last message delivered, once a message is lost (e.g. dropped beyond the pending limits), the heartbeats stop,
the consumer is deleted or the connection is restored. The consumers are created on `consume()` and deleted when the subscription completes; a following `consume()` continues after the last message.

```php
<?php

declare(strict_types=1);

require_once __DIR__ . '/vendor/autoload.php';

use Thesis\Nats;
use Thesis\Nats\JetStream;
use Thesis\Nats\JetStream\Api;

$nc = new Nats\Client(Nats\Config::default());
$js = $nc->jetStream();

$consumer = $js->orderedConsumer('Orders', new JetStream\OrderedConsumerConfig(
filterSubjects: ['orders.created'],
deliverPolicy: Api\DeliverPolicy::ByStartSequence,
startSeq: 100,
));

$subscription = $consumer->consume(
static function (JetStream\Delivery $delivery): void {
dump($delivery->metadata?->streamSequence, $delivery->message->payload);
},
);

// or read them from an iterator
$consumer->stop();

foreach ($consumer->consumeIterator() as $delivery) {
dump($delivery->message->payload);
}
```

`Stream::orderedConsumer()` does the same for a stream at hand. The heartbeat interval (5 seconds by default), the inactive threshold of the server consumers,
the number of reset attempts before the subscription fails (unlimited by default) and the pending limits are set through `OrderedConsumerConfig`.
A consumer is consumed by one subscription at a time.

#### Get message

```php
Expand Down
108 changes: 74 additions & 34 deletions src/Client.php
Original file line number Diff line number Diff line change
Expand Up @@ -289,7 +289,7 @@ public function subscribeIterator(
pendingLimits: $pendingLimits,
);

return self::iterate($queue, $subscription);
return Iter\PipelineIterator::fromSubscription($queue, $subscription);
}

/**
Expand Down Expand Up @@ -410,7 +410,7 @@ public function requestMany(
throw $e;
}

return self::iterate($queue, $subscription);
return Iter\PipelineIterator::fromSubscription($queue, $subscription);
}

/**
Expand Down Expand Up @@ -715,6 +715,72 @@ private function register(
return $subscription;
}

/**
* Subscribes one more subject into an active subscription: its messages go to the same handler, in turn with the others
* and under the same pending limits. E.g. an ordered consumer switches its deliveries to a new subject without
* replacing the subscription it handed out, while the server counts only a literal subject as an interest.
* A JetStream delivery carries the subject of the stored message, not the one subscribed, so $onReceive tells
* which deliveries came through this subject: it is called with each of them before it is queued.
*
* @internal
* @param non-empty-string $subject
* @param \Closure(Delivery): void $onReceive
* @return \Closure(?Cancellation=): void unsubscribes the subject
* @throws NatsException
* @throws \Throwable
*/
public function attach(
Subscription $subscription,
string $subject,
\Closure $onReceive,
?Cancellation $cancellation = null,
): \Closure {
$push = null;
foreach ($this->subscribers as $subscriber) {
if ($subscriber->subscription === $subscription) {
$push = $subscriber->push;
break;
}
}

if ($push === null) {
throw new \LogicException('The subscription is not active.');
}

$subscriptionId = $this->subscriptionIdGenerator->nextId();

// Without the subscription: it is completed through its own subject only.
$this->subscribers[$subscriptionId] = new Subscriber(
subject: $subject,
queueGroup: null,
push: static function (Delivery $delivery) use (
$push,
$onReceive,
): bool {
$onReceive($delivery);

return $push($delivery);
},
);

try {
$this->sub($subject, $subscriptionId, cancellation: $cancellation);
} catch (\Throwable $e) {
unset($this->subscribers[$subscriptionId]);

throw $e;
}

$client = \WeakReference::create($this);

return static function (?Cancellation $cancellation = null) use (
$client,
$subscriptionId,
): void {
$client->get()?->unsubscribe($subscriptionId, $cancellation);
};
}

/**
* @param non-empty-string $subject
* @param callable(Delivery): void $handler
Expand Down Expand Up @@ -921,7 +987,12 @@ private function failSubscriptions(Protocol\Err $error, \Throwable $e): bool
return $failed;
}

private function reportError(\Throwable $e): void
/**
* Reports an error that does not end anything to the {@see self::onError()} callbacks, e.g. one of a JetStream consumer.
*
* @internal
*/
public function reportError(\Throwable $e): void
{
$this->logger->warning('Nats error: {message}', ['message' => $e->getMessage(), 'exception' => $e]);

Expand Down Expand Up @@ -1022,37 +1093,6 @@ private function closed(Connection\Connection $connection, ?\Throwable $error):
}
}

/**
* @param Pipeline\Queue<Delivery> $queue
* @return Iterator<Delivery>
*/
private static function iterate(Pipeline\Queue $queue, Subscription $subscription): Iterator
{
// The iterator ends the way the subscription does: cleanly after stop() or drain(), with the error
// after a failure, e.g. the connection closed by the server. Otherwise the consumer would wait forever.
$future = async(static function () use (
$queue,
$subscription,
): void {
try {
$subscription->awaitCompletion();
} catch (\Throwable $e) {
if (!$queue->isComplete()) {
$queue->error($e);
}

return;
}

if (!$queue->isComplete()) {
$queue->complete();
}
});
$future->ignore();

return Iter\PipelineIterator::fromQueue($queue, $subscription);
}

/**
* Awaits the future, returning null if it has failed: the failure has already been reported
* to the caller who initiated the operation, and there is nothing to clean up.
Expand Down
40 changes: 40 additions & 0 deletions src/Exception/ConsumerSequenceMismatch.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
<?php

declare(strict_types=1);

namespace Thesis\Nats\Exception;

use Thesis\Nats\Client;
use Thesis\Nats\NatsException;

/**
* A push consumer has delivered messages the subscription never received, e.g. dropped as a slow consumer:
* the idle heartbeat names a later consumer sequence than the last one received. Reported to {@see Client::onError()},
* the subscription keeps going; the lost messages can be read again starting at {@see $streamResumeSequence}.
*
* @api
*/
final class ConsumerSequenceMismatch extends NatsException
{
/**
* @param non-negative-int $consumerSequence the consumer sequence of the last message received
* @param non-negative-int $lastConsumerSequence the consumer sequence of the last message the server delivered
* @param positive-int $streamResumeSequence the stream sequence right after the last message received
*/
public function __construct(
public readonly string $stream,
public readonly string $consumer,
public readonly int $consumerSequence,
public readonly int $lastConsumerSequence,
public readonly int $streamResumeSequence,
) {
parent::__construct(\sprintf(
'Consumer "%s" of the stream "%s" delivered up to the consumer sequence %d, the last one received is %d; resume from the stream sequence %d.',
$consumer,
$stream,
$lastConsumerSequence,
$consumerSequence,
$streamResumeSequence,
));
}
}
27 changes: 27 additions & 0 deletions src/Header/LastConsumer.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
<?php

declare(strict_types=1);

namespace Thesis\Nats\Header;

/**
* Sent with an idle heartbeat of a consumer: the consumer sequence of the last message the consumer delivered.
*
* @api
*/
final readonly class LastConsumer
{
private const string HEADER = 'Nats-Last-Consumer';

/**
* @return ScalarKey<non-negative-int>
*/
public static function header(): ScalarKey
{
/**
* @var ScalarKey<non-negative-int>
* @phpstan-ignore varTag.type
*/
return ScalarKey::int(self::HEADER);
}
}
Loading
Loading