multi-actor worker API by BenTheElder · Pull Request #1283 · agent-substrate/substrate

Benjamin Elder (BenTheElder) · 2026-08-28T05:14:23Z

Part of #1266

This is a draft of the core API + data store changes.

It's still a large PR, apologies.

The "as rows" commit could be split out, but this takes it to ~all of the breaking changes we can't hide behind updating internals.

Same for the claimlock, but in both cases it seems these are worth understanding when considering the API shape.

They're loadbearing for performance once we actually have multi-actor workers.

Benjamin Elder (BenTheElder) · 2026-08-28T05:49:49Z

x-ref: #1164 🙈

Tim Hockin (thockin)

I focused on API stuff. The atomic update of an assignment and the worker seems like something that will hurt us later, but maybe workers just not updated often enough for that to matter?

It's tricky logic and could do with more explanation (and specifically the "not updated often enough to matter" rational).

I did not focus on validation yet

Tim Hockin (thockin) · 2026-08-30T19:49:57Z

-// declared resource limits. An unset message, or a zero field within it, means
-// "unknown/unset" for that dimension: treated as unconstrained so placement is
-// not blocked (matching the pre-capacity behavior).
+// WorkerCapacity is what a worker pod has to give the Actors it hosts, and also


This comment is also unclear. I think you should clarify a resource model but "give/take" wording is sort of clunky.

IIUC this is the total capacity of the worker, independent of what may currently be allocated to actors. Is that right? So "available" is "worker.capacity - worker.status.allocated".

How would we model overcommit? Worker lies about capacity? Maybe add an overcommit scalar (e.g. 1.5)?

How would we model overcommit? Worker lies about capacity? Maybe add an overcommit scalar (e.g. 1.5)?

My gut take was worker lies about capacity.

We will have to filter this stuff up to autoscaling, that approach seems easy enough to reason about but perhaps too limiting.

We plan to consolidate this with status.

Benjamin Elder (BenTheElder) · 2026-08-31T21:09:29Z

The atomic update of an assignment and the worker seems like something that will hurt us later, but maybe workers just not updated often enough for that to matter?

FWIW I've pushed 28k claims/s with a test of the storage layer, it scales pretty well with cores.. For overcomitted toy actors, 97k in 60s ... mostly bound by dataplane. This is only going to hurt us if the rest of the system gets really fast or if we have reason to do a lot of other writes to workers unrelated to scheduling. We're by far dataplane bound even at fairly considerable scale with this.

We could do something else like ... a distributed scheduling architecture, but I'm not sure that's necessary.

I've realized some other defects in this as-is when I got some time to sit and think over the weekend. Working on it.

I'd really like registered capacity to be orthogonal to pod creation. The worker should self report CPU and memory, even if we initially just downward API that in from the pods. That's a minor tweak (we already do this for actor count) but it needs some work.
... devices. They're under-modeled currently, but they do sort of work in main. This regresses. Not straighforward.

Benjamin Elder (BenTheElder) · 2026-09-01T04:00:26Z

I will iterate on this a bit more, including some cleanup to the commits etc, and notify when it's ready.

For now, logging off.

Eitan Yarmush (EItanya)

Overall this makes sense, I still need to mull the whole thing over, but I am really liking the direction.

Codex found a couple potential races that we can confirm:

  Binding commits before the Actor is updated to reference the Worker. If ateapi crashes in
  between, the Worker holds an assignment that the Actor does not know about.

  Later, DeleteActor skips Worker cleanup when actor.status.worker_assignment is empty. There is
  no assignment-to-Actor foreign key or reconciler, so the capacity can remain leaked until the
  Worker disappears.

  Smallest sound fix: deletion must look up and release an assignment by Actor UID even when the
  Actor lacks a backlink. Add a test covering bind → missing Actor update → delete.

  When an Actor is already assigned to the same Worker, BindActorToWorker subtracts the old
  resources and adds the replacement, but does not call admit.

  Because Actor.actor_template is mutable, a stale claim can be resumed with larger limits and
  push the Worker over capacity.

  Smallest fix: subtract the previous reservation, run admission against that effective Worker,
  then add the replacement.

Eitan Yarmush (EItanya) · 2026-09-02T20:04:16Z

+  // The Worker being reported on. atespace is always empty; Workers are
+  // global-scoped.
+  // +k8s:opaqueType
+  ObjectRef worker = 1;


Doesn't this need to be inferred from the certificate anyway, is this redundant?

We don't currently dial directly from the worker, it comes from ateom => atelet.

I have confidence that will change in the future but ... not sure when. Probably after we actually have multi-actor worker ... bit chicken and egg.

Eitan Yarmush (EItanya) · 2026-09-02T20:13:52Z

  // +k8s:optional
-  // +k8s:minimum=1
-  int64 cpu_milli = 1;
+  Resources resources = 1;


Feel free to throw this away. If we parse this into a map in go anyway, should it be a map in the proto as well?

Tim Hockin (@thockin) had an earlier comment about reusing the Resources type. I'm on the fence. Later we probably need to model more things. I don't love the Resources type but I like that it matches what we have on ActorTemplate.

So I lean towards keeping it for now. If we want to use cpu_milli / memory bytes consistently we should rewrite ActorTemplate too.

I don't love Resources but it maps to k8s. There are ...issues... properly handling quantity strings, which I am chasing down on the k8s side. I think we should ultimately decide to either support quantity strings all over or enforce quantized int values (e.g. cpu_millicores, memory_mb) all over.

I suggest we bulk edit that in a follow-up, it's somewhat orthogonal and I'm not totally sold either way yet.

We're also going to need more complex resource types O(soon) to express GPUs properly, a count really isn't sufficient.

Added to #1378

Benjamin Elder (BenTheElder) · 2026-09-02T23:25:16Z

Codex found a couple potential races that we can confirm:

These are real and should be fixed.

I need to do some other remaining cleanup.

Tim Hockin (thockin)

Proto seems straight forward at this point

Tim Hockin (thockin) · 2026-09-02T23:48:15Z

+  //
+  // +k8s:optional
+  // +k8s:minimum=1
+  int32 page_size = 2;


Julian Gutierrez Oschmann (@juli4n) Should we bundle these into a generic ListOptions ?

Obviously for a followup PR

I don't have a strong opinion. We can wrap it and reuse the validations. Each response still needs to define a next page token.

Tim Hockin (thockin) · 2026-09-03T00:38:26Z

  // +k8s:optional
-  // +k8s:minimum=1
-  int64 cpu_milli = 1;
+  Resources resources = 1;


I don't love Resources but it maps to k8s. There are ...issues... properly handling quantity strings, which I am chasing down on the k8s side. I think we should ultimately decide to either support quantity strings all over or enforce quantized int values (e.g. cpu_millicores, memory_mb) all over.

Tim Hockin (thockin) · 2026-09-03T00:40:05Z

+  // What a Worker supplies, named as an ActorTemplate names what it asks for,
+  // so the two are one vocabulary and subtract directly. A name the Worker does
+  // not report is unconstrained rather than absent, so a Worker that has said
+  // nothing is not unschedulable.


I don't think this semantic works -- if I ask for 128 cores, you can't put me somewhere unless you know I have them. If I ask for GPU, you can't put me on a worker that has no GPUs.

Agree ...

Aside: this is also a meh claude comment for sure.

my main takeaway from this PR is to cut off claude for API changes and stick to boilerplate fixes. even writing out what you want winds up with ... ehh not there yet.

next time I will entirely hand-craft the API commit at minimum.

I think what I want here is to require that these are set, and make ateom fully set them now.

Tim Hockin (thockin) · 2026-09-03T01:08:11Z

+// Actor is placed only where free covers its limits in every dimension.
+//
+// A Worker overcommits by reporting more than it has.
 message WorkerCapacity {


Do we need this intermediate struct if we use Resources?

I'm not sure if it will just be resources, and actor count is not a dimension we represent elsewhere.

Elaborating: I think we're going to need a more complex model for devices than <name: count>. So we'd have to add a new type anyhow.

Benjamin Elder (BenTheElder) · 2026-09-03T04:55:06Z

7d9ced4 is substantial new code hoisted to this branch to manage #1283 (comment) -- we have the ateoms fully self-report capacity now.

For now it's just count = 1 and cpu/memory from downward API, to keep complexity down and behavior close to before (but still only after ateom boots).

In the future we will report live capacity (read cgroups instead, consider IPPR), account for overcommit, etc.

Benjamin Elder (BenTheElder) · 2026-09-03T05:21:57Z

 // to the neutral badge, which is the right visual treatment.
 func workerPhase(w *ateapipb.Worker) string {
-	if w.GetStatus().GetAllocated().GetActors() > 0 {
+	if w.GetStatus().GetAllocation().GetAllocated().GetActors() > 0 {


a little clunky but oh well

Benjamin Elder (BenTheElder) · 2026-09-03T06:29:56Z

I think all review comments are addressed. 💤

Eitan Yarmush (EItanya)

Mostly nits, but also a couple real potential races

Eitan Yarmush (EItanya) · 2026-09-03T11:58:27Z

+  // mixed ateom versions.
+  //
+  // atelet calls this with its own client certificate, as it does for
+  // MintCert. Idempotent: re-sending the same capacity is not a write.


Idempotent: re-sending the same capacity is not a write.

Is this trying to signal that it won't kick off a watch event for that worker?

Yes. reworded.

Eitan Yarmush (EItanya) · 2026-09-03T12:01:54Z

+
+// retryReport calls send until it succeeds or ctx ends, backing off between
+// attempts.
+func retryReport(ctx context.Context, send func() error, backoff time.Duration) error {


super nit: this function takes a callback to be generic, (and maybe for the unit tests), but makes reading it slightly less clear as it's only actually retrying the reportOnce. Consider just getting rid of the callback and call directly.

Eitan Yarmush (EItanya) · 2026-09-03T12:04:43Z

+// worker reports once, so an accepted call is the only thing that puts
+// capacity on the Worker, and a Worker record the syncer has not created yet
+// is the ordinary reason for a first attempt to fail.
+func (s *workerCapacityService) SetWorkerCapacity(ctx context.Context, req *ateletpb.SetWorkerCapacityRequest) (*ateletpb.SetWorkerCapacityResponse, error) {


Are we doing this ateom -> atelet -> ateapi because that's what we already have, or because that's what we actually want?

The former. But we could switch in the future with only authz changes. I'm not sure we want to introduce "ateom uses the network directly" in this PR.

Eitan Yarmush (EItanya) · 2026-09-03T12:10:31Z

+// writeAndAppendEventFor is writeAndAppendEvent for a write whose event
+// payload is only known once the transaction has read what it is changing.
+// fn returns the worker to announce, or nil for a write that changed nothing.
+func (p *Persistence) writeAndAppendEventFor(ctx context.Context, eventType store.WorkerEventType, fn func(ctx context.Context, tx pgx.Tx) (*ateapipb.Worker, error)) error {


These helpers appear identical except that writeAndAppendEventFor discards the Worker result. The existing helper already supports discovering the event payload inside the transaction. Could Bind ignore its returned Worker and Release return it directly, then remove this duplicate?

Eitan Yarmush (EItanya) · 2026-09-03T12:46:34Z

 // belongs to the caller, which knows whether that is the state it wanted.
 func (w *WorkerWorkflow) DeleteWorker(ctx context.Context, name string, pre store.DeletePreconditions) (*ateapipb.Worker, error) {
 	worker, err := w.loadWorkerForDelete(ctx, name)
 	if err != nil {


Codex pointed out something interesting here. Essentially Worker remains schedulable during this deletion run.

A Worker remains ACTIVE while this sweep runs, so a concurrent resume can bind a new Actor after its page has been processed. finalizeDeleted then deletes the Worker and cascades that new assignment without clearing the Actor's backlink. The worker syncer supplies no version precondition, so the bind's Worker version bump does not prevent this. Could deletion first persist a state that makes Applies reject new binds, then sweep assignments?

should be fixed

status.assignment held a single Actor. It becomes repeated, reusing field 2 so an old record reads back as a set of one, and status.allocated carries the running total that placement reads for every Worker on every decision. WorkerCapacity now holds a Resources set, named the way an ActorTemplate names its limits so that the two subtract, alongside an actors ceiling for the costs that cpu and memory do not cover. ActorAssignment.resources records what an Actor was admitted for. Capacity can now change over a Worker's life, but it cannot be cleared. A Worker reports its own capacity through the new WorkerReporting service. That is separate from Control because it is served only to atelet, and only for the Workers on its own node.

Readers use resources.WorkerAssignmentFor, and binding and releasing update status.allocated alongside the assignment list. Accounting moves to named quantities in resources.Quantities, which emits them in sorted order. Resources is a repeated field, so two totals that differ only in order compare unequal and rewrite the record for no change. Placement compares what is free in each dimension, and eligibility is asked separately from room. A caller re-validating a Worker that already holds the Actor would otherwise be told there is no space and evict it. A Worker whose recorded capacity does not parse is treated as full. Clearing capacity is rejected by a hand-written check. Declarative validation can mark a field immutable, but it cannot allow a field to change while still requiring it to be set. ListWorkers omits the assignments and reports occupancy through allocated. kubectl-ate prints ASSIGNED(n/m).

atelet reports on behalf of the Workers on its node, authenticated by its own client certificate as it is for MintCert. A Worker on another node comes back as NOT_FOUND. The atelet authentication moves into a package both services share. Capacity is the Worker's to report rather than the control plane's to infer, so the syncer no longer derives cpu and memory from the pod. Until a report arrives, the ceiling CreateWorker reifies holds the Worker to one Actor. A reporter can leave out dimensions it does not know, and those keep whatever is already recorded. Re-sending an unchanged capacity does not write. Nothing calls this yet.

A Worker carried its assignments inside its own record, so the record, its change event, and every watcher's copy of it all grew with the number of Actors on it. Each assignment becomes a row of its own. status.allocated stays on the Worker, because placement reads it for every candidate. The store updates it in the same transaction as the assignment it counts. Binding does not first read the assignment it might be replacing. The insert landing is proof that nothing was there, and a read could not see a claim that commits after it.

Claiming a worker rewrote its whole record, so goroutines racing on the same worker refused one another: 21% of activations failed at 12 in flight, and 67% at 24. BindActorToWorker now takes an admit callback and asks inside the transaction that holds the worker's row lock, so the answer cannot go stale before the write, and a refusal rolls the speculative row back. That removes the version precondition, the fresh read, and the assignment lookup. The retry budget had to grow, because a claim refused for want of room re-runs scheduling.

Capacity is observed, not requested: the Worker reports it through SetWorkerCapacity and no client writes it. Move it from the Worker to WorkerStatus, beside the allocated total it is compared against. UpdateWorker restores the stored status, so a client can no longer change or clear capacity at all. The hand-written no-clearing check goes with it. Drops the reserved field numbers, which nothing shipped needs, and the opaqueType markers on SetWorkerCapacityRequest -- those cancel the validation of the type they annotate, which is the opposite of what this RPC wants.

Capacity read an absent dimension as unconstrained, so an Actor asking for a GPU could be placed on a Worker that never reported one. Absent now means none of it: a Worker reports everything it can supply, and a name missing from its capacity is one no Actor may ask it for. SetWorkerCapacity therefore replaces rather than merges. The merge existed to stop a reporter that knew only its actor ceiling from erasing compute it could not speak to, which a complete report removes -- and replacing is what the name says.

CreateWorker was fabricating a capacity of one actor for every Worker it stored. That contradicts the rule the API just adopted -- a Worker reports every dimension it has, so an absent one is none of it -- because the control plane was answering on the Worker's behalf before the worker had said anything. Nothing was calling SetWorkerCapacity, so add the reporter that makes it real: ateom answers a new GetCapacity, atelet sweeps the node's ateoms and forwards what they say, and a Worker holds no capacity until that lands. ateom learns its compute limits from the downward API, since it has no reader for its own cgroup and inventing one is a larger change. Left as a TODO: the environment is fixed at pod creation, so it goes stale under in-place pod resize.

Capacity and allocated are only ever read together -- placement subtracts one from the other -- so they sit under a single status.allocation rather than as two sibling fields. WorkerCapacity becomes WorkerResources: the same message carries both totals, so naming it after one of them was wrong.

Two problems Codex found in the assignment subresource. A rebind marshaled the metadata a first bind gets, so a retried claim replaced the recorded assignment with a fresh uid, create time and version 1, making an update look like a new subresource. It now carries the recorded identity forward. DeleteWorker swept a Worker that was still ACTIVE, so a resume could bind an Actor to a page already passed and the delete would cascade that assignment away, leaving the Actor pointing at a Worker that is gone. The delete drains first. The caller's guards are checked before that, against the Worker it observed, since the drain moves the version. Also drops writeAndAppendEventFor, which differed from writeAndAppendEvent only in discarding the returned Worker.

Tim Hockin (thockin)

Just the proto file again

Tim Hockin (thockin) · 2026-09-03T16:04:30Z

+  //
+  // +k8s:optional
+  // +k8s:minimum=1
+  int32 page_size = 2;


Obviously for a followup PR

Tim Hockin (thockin) · 2026-09-03T18:41:20Z

  // +k8s:optional
-  // +k8s:minimum=1
-  int64 cpu_milli = 1;
+  Resources resources = 1;


Added to #1378

ReleaseActorFromWorker took an expectedVersion it never read, so callers passed a guard that did nothing. Honoring it would be wrong rather than a fix: a Worker's version moves whenever any of its Actors binds or releases, so on a Worker holding thousands the guard would refuse releases for changes that have nothing to do with the Actor being released, and the delete sweep would fail from its second page on. The assignment's own key and the Worker's row lock are what make the release exact.

The Go handler, its test and the apitool exemption follow the rename.

Tim Hockin (thockin)

This PR is big enough that there are no obvious bugs. I'm still worried about the multi-resource "dance" of creating the assignment. There are things I am not quite sure are right but are not obviously wrong either - for example a RESUMING actor bound to a DRAINING worker currently crashes but it's not clear to me if that is AI paranoia, if it is a legitimate "oh no" situation, or if it is actually an acceptable corner case.

Basically I'm trying to convince myself that the high-level flow is right. There are long sequences of tricky, conditional-rich code with no clear comments on assumed preconditions or asserted postconditions.

I'm still walking thru it and I will send you a diff in bit just for some comments I think could be better. I don't see a reason to block this going in at this point.

Tim Hockin (thockin) · 2026-09-03T20:13:17Z

+//
+// An absent name is none of that resource. A Worker reports every dimension it
+// has, so a name missing from its capacity is one it cannot supply at all.
+type Quantities map[string]resource.Quantity


OK for the moment, but we need to decide if we REALLY want to use Quantity the string format and if so if we really want to parse it into Quantity the type or into a math/big.Float or something

Yeah. Do we already have a TODO for this? Should we file one?

…downward API files

Benjamin Elder (BenTheElder) added area/api User-facing API changes area/node labels Aug 28, 2026

Benjamin Elder (BenTheElder) force-pushed the worker-assignments-as-rows branch 2 times, most recently from 96e9c63 to 975778f Compare August 28, 2026 23:58

Benjamin Elder (BenTheElder) requested review from Julian Gutierrez Oschmann (juli4n) and Tim Hockin (thockin) August 29, 2026 02:12

Benjamin Elder (BenTheElder) force-pushed the worker-assignments-as-rows branch 3 times, most recently from a7b750e to 0533447 Compare August 29, 2026 21:21

Tim Hockin (thockin) reviewed Aug 30, 2026

View reviewed changes

Benjamin Elder (BenTheElder) mentioned this pull request Sep 1, 2026

Select devices per-container in the ActorTemplate #752

Open

Benjamin Elder (BenTheElder) force-pushed the worker-assignments-as-rows branch 2 times, most recently from 7e01bfb to 725a108 Compare September 1, 2026 02:29

Joe Betz (jpbetz) mentioned this pull request Sep 1, 2026

ateapi: add versioned PostgreSQL schema migrations #1196

Merged

Benjamin Elder (BenTheElder) force-pushed the worker-assignments-as-rows branch 3 times, most recently from 4644d42 to cd33f4d Compare September 2, 2026 03:31

Benjamin Elder (BenTheElder) marked this pull request as ready for review September 2, 2026 03:38

Benjamin Elder (BenTheElder) requested a review from Tim Hockin (thockin) September 2, 2026 03:38

Benjamin Elder (BenTheElder) force-pushed the worker-assignments-as-rows branch from cd33f4d to 4e67536 Compare September 2, 2026 18:24

Eitan Yarmush (EItanya) reviewed Sep 2, 2026

View reviewed changes

Benjamin Elder (BenTheElder) force-pushed the worker-assignments-as-rows branch from 4e67536 to bfcd462 Compare September 2, 2026 23:24

Benjamin Elder (BenTheElder) force-pushed the worker-assignments-as-rows branch from bfcd462 to 787f032 Compare September 3, 2026 00:00

Benjamin Elder (BenTheElder) commented Sep 3, 2026

View reviewed changes

Comment thread pkg/proto/ateapipb/ateapi.proto

Benjamin Elder (BenTheElder) force-pushed the worker-assignments-as-rows branch from 787f032 to e56db3c Compare September 3, 2026 01:10

Tim Hockin (thockin) reviewed Sep 3, 2026

View reviewed changes

Benjamin Elder (BenTheElder) force-pushed the worker-assignments-as-rows branch from e56db3c to 15394e9 Compare September 3, 2026 01:27

Julian Gutierrez Oschmann (juli4n) reviewed Sep 3, 2026

View reviewed changes

Comment thread cmd/ateapi/internal/workercache/workercache.go Outdated

Julian Gutierrez Oschmann (juli4n) reviewed Sep 3, 2026

View reviewed changes

Comment thread tools/apitool/internal/lint/list_test.go Outdated

Benjamin Elder (BenTheElder) force-pushed the worker-assignments-as-rows branch from 15394e9 to 42628d0 Compare September 3, 2026 04:20

Benjamin Elder (BenTheElder) commented Sep 3, 2026

View reviewed changes

Benjamin Elder (BenTheElder) force-pushed the worker-assignments-as-rows branch 2 times, most recently from dbee797 to 2360665 Compare September 3, 2026 05:37

Eitan Yarmush (EItanya) reviewed Sep 3, 2026

View reviewed changes

Benjamin Elder (BenTheElder) added 10 commits September 3, 2026 11:26

Benjamin Elder (BenTheElder) force-pushed the worker-assignments-as-rows branch from f5c0685 to 5886f4b Compare September 3, 2026 18:34

Tim Hockin (thockin) reviewed Sep 3, 2026

View reviewed changes

Julian Gutierrez Oschmann (juli4n) reviewed Sep 3, 2026

View reviewed changes

Comment thread pkg/proto/ateapipb/ateapi.proto Outdated

Benjamin Elder (BenTheElder) added 3 commits September 3, 2026 13:24

less claude comments

263bd7c

ListWorkerActorAssignments and actor_assignments

32f8d8c

The Go handler, its test and the apitool exemption follow the rename.

Tim Hockin (thockin) reviewed Sep 3, 2026

View reviewed changes

Benjamin Elder (BenTheElder) added 2 commits September 3, 2026 15:32

ateapi: squash worker assignments into initial schema

b092b08

refactor: switch ateom capacity limits from environment variables to …

ae470b1

…downward API files

Uh oh!

Conversation

Benjamin Elder (BenTheElder) commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Benjamin Elder (BenTheElder) commented Aug 28, 2026

Uh oh!

Tim Hockin (thockin) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Benjamin Elder (BenTheElder) commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Benjamin Elder (BenTheElder) commented Sep 1, 2026

Uh oh!

Eitan Yarmush (EItanya) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Benjamin Elder (BenTheElder) Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Benjamin Elder (BenTheElder) Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Benjamin Elder (BenTheElder) commented Sep 2, 2026

Uh oh!

Uh oh!

Tim Hockin (thockin) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Benjamin Elder (BenTheElder) Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Benjamin Elder (BenTheElder) commented Aug 28, 2026 •

edited

Loading

Benjamin Elder (BenTheElder) commented Aug 31, 2026 •

edited

Loading

Benjamin Elder (BenTheElder) Sep 2, 2026 •

edited

Loading

Benjamin Elder (BenTheElder) Sep 2, 2026 •

edited

Loading

Benjamin Elder (BenTheElder) Sep 3, 2026 •

edited

Loading