Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
126 changes: 126 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,9 @@ Pure non-blocking (fiber based) strictly typed full-featured PHP driver for NATS
- [Mirrors and sources](#mirrors-and-sources)
- [NATS ObjectStore](https://docs.nats.io/nats-concepts/jetstream/obj_store)
- [Store objects](#store-objects-in-the-buckets)
- [List and update objects](#list-and-update-objects)
- [Files and streams](#files-and-streams)
- [Object Store status](#object-store-status)
- [Watch Object Store](#watch-object-store)
- [NATS CRDT](#nats-crdt)
- [Add Counter](#add-counter)
Expand Down Expand Up @@ -1449,6 +1452,129 @@ $nc->drain();
- If the reader or a publish fails in the middle of `put()`, the chunks already published are purged and the error is rethrown; the previous version of the object stays as it was.
- `get()` reads the chunks through a temporary consumer, deleted after the object is read to the end. It throws `Nats\Exception\ObjectIsInvalid` at once if the stream holds fewer chunks than the meta tells. The size and the SHA-256 digest of the chunks are checked against the meta: on a mismatch, or when the chunks disappear while reading, the read of the stream throws `Nats\Exception\ObjectIsInvalid`.
- `info()->mtime` is the time the server stored the meta, with sub-second precision.
- `info()` returns null for a deleted object, `info($name, showDeleted: true)` returns its meta with `deleted: true`.

#### List and update objects

```php
<?php

declare(strict_types=1);

require __DIR__ . '/vendor/autoload.php';

use Thesis\Nats;
use Thesis\Nats\JetStream\ObjectStore\ObjectMeta;
use Thesis\Nats\JetStream\ObjectStore\StoreConfig;

$nc = new Nats\Client(Nats\Config::default());
$js = $nc->jetStream();

$store = $js->createOrUpdateObjectStore(new StoreConfig(
store: 'code',
));

$store->put(new ObjectMeta('app.php'), '<?php echo 1;');
$store->put(new ObjectMeta('old.php'), '<?php echo 2;');
$store->delete('old.php');

foreach ($store->list() as $info) {
dump("{$info->name}: {$info->size} bytes, changed at {$info->mtime->format('c')}");
}

dump(count($store->list(showDeleted: true))); // 2

$store->updateMeta('app.php', new ObjectMeta(
name: 'main.php',
description: 'The entrypoint',
metadata: ['owner' => 'core'],
));

dump((string) $store->get('main.php'));

$nc->drain();
```

- `list()` returns the metas of the objects (links included) in the order they were last changed, an empty list for an empty store. It reads them through a temporary consumer, deleted afterwards. A part of the list is never returned: if the client is stopped or the connection is lost before every meta is received, `list()` throws `Nats\Exception\ObjectListIsIncomplete`.
- `updateMeta()` changes the name, the description, the headers and the metadata, the data is not copied. The chunk size of the new meta is ignored. It throws `Nats\Exception\ObjectNotFound` for a missing or deleted object and `Nats\Exception\ObjectAlreadyExists` when the new name belongs to another object; a deleted object does not hold its name.
- `addLink()` throws `Nats\Exception\ObjectAlreadyExists` when the name belongs to an object that is not a link.

#### Files and streams

```php
<?php

declare(strict_types=1);

require __DIR__ . '/vendor/autoload.php';

use Amp\ByteStream\ReadableResourceStream;
use Thesis\Nats;
use Thesis\Nats\JetStream\ObjectStore\ObjectMeta;
use Thesis\Nats\JetStream\ObjectStore\StoreConfig;

$nc = new Nats\Client(Nats\Config::default());
$js = $nc->jetStream();

$store = $js->createOrUpdateObjectStore(new StoreConfig(
store: 'backups',
));

$store->putFile('/var/backups/db.sql', new ObjectMeta('db.sql'));
$store->getFile('db.sql', '/tmp/db.sql');

// Any amphp byte stream: a socket, a process output, another object.
$store->put(new ObjectMeta('stdin'), new ReadableResourceStream(STDIN));
$object = $store->get('db.sql') ?? throw new \RuntimeException('db.sql is not found.');
$store->put(new ObjectMeta('db-copy.sql'), $object->stream);

$nc->drain();
```

- `putFile()` stores the file under its path, like nats.go, unless a meta is given.
- `getFile()` writes the object to a temporary file next to the target and renames it over the target, so the file is replaced only by a complete, verified object. It returns null if there is no such object; if reading the object fails, e.g. on a digest mismatch, the temporary file is removed, an existing file is left untouched, and the error is rethrown.
- A file that cannot be opened or is not a regular file throws `Nats\Exception\FileIsNotAccessible`; so does a read error of a `ResourceReader`, and nothing is stored. A failing `ReadableStream` throws its own `Amp\ByteStream\StreamException`.
- `put()` accepts a string, a `Reader` (`StringReader`, `ResourceReader`, `StreamReader`) or an `Amp\ByteStream\ReadableStream`. The parts of a stream are glued, so that every chunk but the last one has the chunk size.

#### Object Store status

```php
<?php

declare(strict_types=1);

require __DIR__ . '/vendor/autoload.php';

use Thesis\Nats;
use Thesis\Nats\JetStream\ObjectStore\StoreConfig;
use Thesis\Time\TimeSpan;

$nc = new Nats\Client(Nats\Config::default());
$js = $nc->jetStream();

$store = $js->createOrUpdateObjectStore(new StoreConfig(
store: 'code',
description: 'Source code',
ttl: TimeSpan::fromDays(7),
));

$status = $store->status();

dump(
$status->bucket, // code
$status->description, // Source code
$status->ttl, // 7 days, null if objects are kept forever
$status->storage, // StorageType::File
$status->replicas, // 1
$status->sealed, // false
$status->size, // bytes of the stored chunks and metas
$status->compressed, // false
$status->metadata,
$status->info, // the stream info behind the store
);

$nc->drain();
```

#### Watch Object Store

Expand Down
12 changes: 12 additions & 0 deletions src/Exception/FileIsNotAccessible.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
<?php

declare(strict_types=1);

namespace Thesis\Nats\Exception;

use Thesis\Nats\NatsException;

/**
* @api
*/
final class FileIsNotAccessible extends NatsException {}
12 changes: 12 additions & 0 deletions src/Exception/ObjectAlreadyExists.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
<?php

declare(strict_types=1);

namespace Thesis\Nats\Exception;

use Thesis\Nats\NatsException;

/**
* @api
*/
final class ObjectAlreadyExists extends NatsException {}
15 changes: 15 additions & 0 deletions src/Exception/ObjectListIsIncomplete.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
<?php

declare(strict_types=1);

namespace Thesis\Nats\Exception;

use Thesis\Nats\NatsException;

/**
* The object store list stopped before every meta was received, e.g. the client was stopped
* or the connection was lost and the metas sent meanwhile are gone.
*
* @api
*/
final class ObjectListIsIncomplete extends NatsException {}
6 changes: 3 additions & 3 deletions src/JetStream/Api/StreamConsumerLimits.php
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,11 @@ public function __construct(
) {}

/**
* @return array<non-empty-string, mixed>
* An object even without limits: the server rejects an empty JSON array, e.g. when a stream config it returned is sent back.
*/
public function jsonSerialize(): array
public function jsonSerialize(): \stdClass
{
return array_filter(
return (object) array_filter(
[
'inactive_threshold' => $this->inactiveThreshold?->toNanoseconds(),
'max_ack_pending' => $this->maxAckPending,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

declare(strict_types=1);

namespace Thesis\Nats\JetStream\KeyValue\Internal;
namespace Thesis\Nats\JetStream\Internal;

use Amp\DeferredFuture;
use Amp\Future;
Expand Down
4 changes: 2 additions & 2 deletions src/JetStream/KeyValue/Bucket.php
Original file line number Diff line number Diff line change
Expand Up @@ -349,8 +349,8 @@ private function startWatch(
$consumer = $this->stream->createOrUpdateConsumer($consumerConfig, $cancellation);

$initialValues = $config->updatesOnly
? Internal\InitialValues::none()
: new Internal\InitialValues($consumer->info->numPending + $consumer->info->delivered->consumerSeq);
? JetStream\Internal\InitialValues::none()
: new JetStream\Internal\InitialValues($consumer->info->numPending + $consumer->info->delivered->consumerSeq);

$subscription = $consumer->push(
new Internal\WatchHandler(
Expand Down
1 change: 1 addition & 0 deletions src/JetStream/KeyValue/Internal/WatchHandler.php
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
use Thesis\Nats\Header;
use Thesis\Nats\JetStream\Delivery;
use Thesis\Nats\JetStream\Internal\IdleListener;
use Thesis\Nats\JetStream\Internal\InitialValues;
use Thesis\Nats\JetStream\KeyValue\Entry;
use Thesis\Nats\JetStream\KeyValue\EntryState;
use Thesis\Nats\Subscription;
Expand Down
96 changes: 96 additions & 0 deletions src/JetStream/ObjectStore/Internal/InfoCollector.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
<?php

declare(strict_types=1);

namespace Thesis\Nats\JetStream\ObjectStore\Internal;

use Amp\DeferredFuture;
use Amp\Future;
use Thesis\Nats\Exception\ObjectListIsIncomplete;
use Thesis\Nats\JetStream\Delivery;
use Thesis\Nats\JetStream\Internal\IdleListener;
use Thesis\Nats\JetStream\Internal\InitialValues;
use Thesis\Nats\JetStream\ObjectStore\ObjectInfo;

/**
* Keeps the metas a list consumer delivers until it has delivered every meta the store had when the list started.
*
* @internal
*/
final class InfoCollector implements IdleListener
{
/** @var list<ObjectInfo> */
public private(set) array $objects = [];

private int $consumerSequence = 0;

/** @var DeferredFuture<never> */
private readonly DeferredFuture $lost;

/**
* @param \Closure(Delivery): ?ObjectInfo $decode
*/
public function __construct(
private readonly \Closure $decode,
private readonly bool $showDeleted,
private readonly InitialValues $initialValues,
) {
/** @var DeferredFuture<never> $lost */
$lost = new DeferredFuture();
$this->lost = $lost;
$this->lost->getFuture()->ignore();
}

/**
* @throws ObjectListIsIncomplete
*/
public function __invoke(Delivery $delivery): void
{
$metadata = $delivery->metadata;
if ($metadata === null || $this->initialValues->delivered()) {
return;
}

// Nothing is acknowledged, so the metas the server sent while the connection was lost are not sent again.
if ($metadata->consumerSequence !== ++$this->consumerSequence) {
throw new ObjectListIsIncomplete("Object metas were lost: expected delivery {$this->consumerSequence}, received {$metadata->consumerSequence}.");
}

$info = ($this->decode)($delivery);

if ($info !== null && ($this->showDeleted || $info->deleted !== true)) {
$this->objects[] = $info;
}

$this->initialValues->receive(max($metadata->pending, 0));
}

/**
* The metas sent while the connection is lost are gone, and the server waits for the flow control reply sent to
* the lost connection, so the list would hang: it fails at once instead.
*/
public function disconnected(): void
{
if (!$this->lost->isComplete() && !$this->initialValues->delivered()) {
$this->lost->error(new ObjectListIsIncomplete('The connection was lost before every object was received.'));
}
}

/**
* Errors once the connection is lost before every meta was received.
*
* @return Future<never>
*/
public function lost(): Future
{
return $this->lost->getFuture();
}

/**
* The server has nothing more to deliver, e.g. the metas were purged after the list had started.
*/
public function idle(): void
{
$this->initialValues->complete();
}
}
23 changes: 23 additions & 0 deletions src/JetStream/ObjectStore/ObjectInfo.php
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,29 @@ public function withTime(\DateTimeImmutable $mtime): self
);
}

/**
* The options, i.e. the link and the chunk size, are kept: they describe the stored data, not the object.
*
* @internal
*/
public function withMeta(ObjectMeta $meta): self
{
return new self(
name: $meta->name,
bucket: $this->bucket,
nuid: $this->nuid,
size: $this->size,
chunks: $this->chunks,
mtime: $this->mtime,
digest: $this->digest,
deleted: $this->deleted,
description: $meta->description,
headers: $meta->headers,
metadata: $meta->metadata,
options: $this->options,
);
}

/**
* @internal
*/
Expand Down
Loading
Loading