Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 71 additions & 7 deletions docs/OPERATION-LOGGING.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,8 +61,11 @@ await runtime.shutdown();
`withRuntimeOperation(name, fn, attributes?, options?)` runs the callback in an always-recorded
process span and emits one completed operation summary (`autter.operation.kind =
"operation"`; request middleware emits `"request"` summaries). `options.from`
continues work from a `runtimeContext.carrier()` (parent link, request id, and an
OTel span link to the producer trace); `options.waitUntil` receives the log flush
continues work handed over from another operation (parent link, request id, and an
OTel span link to the producer trace): it takes a `runtimeContext.carrier()` or
anything `runtimeContext.inject()` produced, see
[Queues, workers and other processes](#queues-workers-and-other-processes);
`options.waitUntil` receives the log flush
promise once the summary is emitted. It returns the callback's
result and rethrows its original error. An unhandled callback error records an
exception in the trace; the log summary is related evidence, not a second issue.
Expand All @@ -86,8 +89,8 @@ the existing `autter.outcome` trace event, including the reporting call site.

`createRuntimeLogger(attributes?)` creates a logger with `debug`, `info`, `warn`,
and `error` methods. `runtimeLogger` is the default instance; `runtimeContext`
exposes the same methods plus `set`, `outcome`, `id`, `requestId`, `fork` and
`carrier` for the current operation from anywhere in its call tree.
exposes the same methods plus `set`, `outcome`, `id`, `requestId`, `fork`,
`carrier` and `inject` for the current operation from anywhere in its call tree.

**Inline messages (1.5.0).** Inside an operation, `debug` and `info` messages are
folded into its summary as `autter.operation.logs` — `{ t, level, message,
Expand All @@ -100,12 +103,73 @@ diagnostic context; use `captureException` for exception grouping or declare a
failed operation outcome for business-failure grouping. Logs inherit the
active operation ID, name, parent operation ID, and application attributes, plus
the active OTel trace/span IDs. Concurrent operations keep separate contexts.
Async context does not cross a queue or process by itself: put
`runtimeContext.carrier()` in the job payload and pass it as
`withRuntimeOperation(name, fn, {}, { from: payload.autter })` in the consumer.
Async context does not cross a queue or process by itself; see
[Queues, workers and other processes](#queues-workers-and-other-processes).
`runtimeContext.fork(name, fn)` links a child to its parent even when the child
outlives it; `runInBackground(name, fn)` does the same without awaiting.

## Queues, workers and other processes

The producer adds the current operation's carrier to what it hands over with
`runtimeContext.inject(target, format?)`; the consumer passes what it received
as `withRuntimeOperation(name, fn, {}, { from })`. The consumer's summary then
has `autter.operation.parent_id` (the producer operation), the producer's
`autter.request.id` and `autter.parent_trace_id`. The consumer's trace is
linked to the producer's (an OTel span link), not continued: it gets its own
trace id, and the request id is what joins the two.

The carrier is `{ v: 1, op, req?, traceparent? }` under the key `autter`: ids
only, never operation attributes, user context or baggage. `inject` returns a
copy and never changes `target`. Outside an operation, or when `target` is not a
plain object or `Headers`, it returns `target` unchanged. It never throws.
`from` ignores anything missing or malformed, and a carrier string over 1,024
characters; the operation then starts unlinked.

| `format` | Where the carrier goes | Consumer passes as `from` |
| --- | --- | --- |
| `"payload"` (default) | `{ ...data, autter: carrier }` | the payload |
| `"headers"` | header `autter: <carrier JSON>` (a `Headers` instance is always treated as headers) | the headers (any case) |
| `"sqs"` | `{ autter: { DataType: "String", StringValue: <carrier JSON> } }` | `MessageAttributes`, or the whole message |

BullMQ (the job payload):

```ts
import { runtimeContext, withRuntimeOperation } from "@autter/runtime-node";

// Producer, inside a request or operation
await queue.add("resize", runtimeContext.inject({ imageId }));

// Worker
new Worker("images", (job) =>
withRuntimeOperation(job.name, () => resize(job.data.imageId), {}, { from: job.data }),
);
```

SQS (a message attribute, so the body stays yours):

```ts
// Producer
await sqs.send(new SendMessageCommand({
QueueUrl,
MessageBody: JSON.stringify(order),
MessageAttributes: runtimeContext.inject({}, "sqs"),
}));

// Consumer: ReceiveMessage needs MessageAttributeNames: ["autter"] (or ["All"]).
// A Lambda SQS record works as is (its messageAttributes are read too).
for (const message of Messages ?? []) {
await withRuntimeOperation("fulfil-order", () => fulfil(JSON.parse(message.Body!)), {}, {
from: message,
});
}
```

Celery (Python): no Autter helper is needed. OpenTelemetry's Celery
instrumentation already continues the trace from the code that sends the task
to the worker that runs it: call `CeleryInstrumentor().instrument()` in both
processes, after `init_tracing()`, as in
[`examples/python/celery`](../examples/python/celery).

## Collection and privacy

- Logs are OTLP/HTTP log records. Both OTLP JSON and protobuf are accepted.
Expand Down
136 changes: 136 additions & 0 deletions packages/runtime-core/src/carrier.ts
Original file line number Diff line number Diff line change
Expand Up @@ -83,3 +83,139 @@ export function parseCarrier(input: unknown): RuntimeCarrier | null {
: {}),
});
}

/** Key (and header / SQS message attribute name) a carrier travels under. */
export const CARRIER_KEY = "autter";

/**
* Where `injectCarrier` puts the carrier:
* - `"payload"` (default): a plain job payload object, as `{ ...payload, autter: carrier }`;
* - `"headers"`: an HTTP header map, as the header `autter: <carrier JSON>`
* (a `Headers` instance is always treated as headers);
* - `"sqs"`: AWS SQS `MessageAttributes`, as
* `{ autter: { DataType: "String", StringValue: <carrier JSON> } }`.
*/
export type CarrierFormat = "payload" | "headers" | "sqs";

// Case-insensitive lookups read at most this many keys of a source object.
const MAX_SCANNED_KEYS = 256;

const hasOwn = (value: object, key: string): boolean =>
Object.prototype.hasOwnProperty.call(value, key);

function isPlainObject(value: unknown): value is Record<string, unknown> {
if (!value || typeof value !== "object" || Array.isArray(value)) return false;
const proto = Object.getPrototypeOf(value);
return proto === Object.prototype || proto === null;
}

function isHeaders(value: unknown): value is Headers {
return typeof Headers !== "undefined" && value instanceof Headers;
}

/**
* Return a copy of `target` that carries `carrier` (see `CarrierFormat`).
* Never mutates `target`: plain objects come back as shallow copies and a
* `Headers` instance as a new `Headers`. Only the carrier's own fields are
* written (`{ v: 1, op, req?, traceparent? }`), never attributes or user
* context. When there is nothing to inject (no valid carrier) or the target
* cannot hold it (not a plain object or `Headers`), `target` is returned
* unchanged. Never throws.
*/
export function injectCarrier<T>(
target: T,
carrier: RuntimeCarrier | null | undefined,
format: CarrierFormat = "payload",
): T {
try {
const valid = carrier ? parseCarrier(carrier) : null;
if (!valid) return target;
const json = JSON.stringify(valid);
if (isHeaders(target)) {
const copy = new Headers(target);
copy.set(CARRIER_KEY, json);
return copy as T;
}
if (!isPlainObject(target)) return target;
if (format === "headers") {
const copy: Record<string, unknown> = {};
for (const key of Object.keys(target))
if (key.toLowerCase() !== CARRIER_KEY) copy[key] = target[key];
copy[CARRIER_KEY] = json;
return copy as T;
}
if (format === "sqs")
return {
...target,
[CARRIER_KEY]: { DataType: "String", StringValue: json },
} as T;
return { ...target, [CARRIER_KEY]: valid } as T;
} catch {
return target;
}
}

/** A carrier value found under the key: JSON string, header list, SQS attribute or object. */
function carrierFromValue(value: unknown): RuntimeCarrier | null {
if (typeof value === "string") return parseCarrier(value);
if (Array.isArray(value))
return value.length === 1 && typeof value[0] === "string"
? parseCarrier(value[0])
: null;
if (!value || typeof value !== "object") return null;
const raw = value as Record<string, unknown>;
// SQS MessageAttributes (SDK: StringValue/DataType; Lambda events: stringValue/dataType).
const stringValue = raw.StringValue ?? raw.stringValue;
if (stringValue !== undefined) {
const dataType = raw.DataType ?? raw.dataType;
return typeof stringValue === "string" &&
(dataType === undefined || (typeof dataType === "string" && dataType.startsWith("String")))
? parseCarrier(stringValue)
: null;
}
return parseCarrier(raw);
}

function valueUnderKey(source: Record<string, unknown>): unknown {
if (hasOwn(source, CARRIER_KEY)) return source[CARRIER_KEY];
// Header maps: names are case-insensitive.
let scanned = 0;
for (const key in source) {
if (++scanned > MAX_SCANNED_KEYS) break;
if (hasOwn(source, key) && key.toLowerCase() === CARRIER_KEY) return source[key];
}
return undefined;
}

function extractFrom(source: unknown, depth: number): RuntimeCarrier | null {
if (typeof source === "string") return parseCarrier(source);
if (isHeaders(source)) return carrierFromValue(source.get(CARRIER_KEY));
if (!source || typeof source !== "object" || Array.isArray(source)) return null;
const raw = source as Record<string, unknown>;
const value = valueUnderKey(raw);
if (value !== undefined) return carrierFromValue(value);
// The carrier itself.
if (hasOwn(raw, "v") && hasOwn(raw, "op")) return parseCarrier(raw);
// A whole SQS message (SDK `MessageAttributes`, Lambda `messageAttributes`).
if (depth === 0) {
const attributes = raw.MessageAttributes ?? raw.messageAttributes;
if (attributes && typeof attributes === "object") return extractFrom(attributes, 1);
}
return null;
}

/**
* Read a carrier from what a consumer received: the carrier itself (object
* or JSON string), a job payload with an `autter` key, HTTP headers (a
* `Headers` instance or a header map; the name is case-insensitive), SQS
* `MessageAttributes`, or a whole SQS message. Validated like `parseCarrier`
* (same size and format limits; invalid optional fields are dropped).
* Returns undefined for anything missing, malformed or oversized. Never throws.
*/
export function extractCarrier(source: unknown): RuntimeCarrier | undefined {
try {
return extractFrom(source, 0) ?? undefined;
} catch {
return undefined;
}
}
4 changes: 4 additions & 0 deletions packages/runtime-core/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -97,10 +97,14 @@ export {
type OtlpResource,
} from "./otlp-logs.js";
export {
CARRIER_KEY,
createCarrier,
extractCarrier,
injectCarrier,
parseCarrier,
formatTraceparent,
parseTraceparent,
type CarrierFormat,
type RuntimeCarrier,
} from "./carrier.js";
export {
Expand Down
113 changes: 113 additions & 0 deletions packages/runtime-core/test/core.test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,9 @@ import {
codeFingerprint,
parseCarrier,
createCarrier,
injectCarrier,
extractCarrier,
CARRIER_KEY,
resolveRequestId,
normalizeRoutePath,
compileIgnore,
Expand Down Expand Up @@ -158,6 +161,116 @@ test("carriers round-trip and reject malformed input", () => {
assert.deepEqual(parseCarrier({ v: 1, op: "op-1", req: "bad id!", traceparent: "junk" }), { v: 1, op: "op-1" });
});

const CARRIER = createCarrier({
op: "6f1c2b9e-1111-4222-8333-944455556666",
req: "req-abcdef12",
traceparent: "00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01",
});

test("injectCarrier/extractCarrier round-trip a job payload under the autter key, without mutating it", () => {
assert.equal(CARRIER_KEY, "autter");
const data = { imageId: 42, sizes: [1, 2] };
const job = injectCarrier(data, CARRIER);
assert.notEqual(job, data, "a shallow copy, not the input");
assert.deepEqual(data, { imageId: 42, sizes: [1, 2] }, "the input is untouched");
assert.deepEqual(job, { imageId: 42, sizes: [1, 2], autter: CARRIER });
assert.equal(job.sizes, data.sizes, "shallow: nested values are shared");
assert.deepEqual(extractCarrier(job), CARRIER);
// Through a queue that serialises payloads as JSON.
assert.deepEqual(extractCarrier(JSON.parse(JSON.stringify(job))), CARRIER);
// A payload whose own fields look like a carrier: the autter key wins.
assert.deepEqual(extractCarrier({ v: 1, op: "resize", autter: CARRIER }), CARRIER);
// The carrier itself, as an object or a JSON string, is accepted too.
assert.deepEqual(extractCarrier(CARRIER), CARRIER);
assert.deepEqual(extractCarrier(JSON.stringify(CARRIER)), CARRIER);
// An existing carrier is replaced, not merged.
const again = injectCarrier({ autter: { v: 1, op: "older" } }, CARRIER);
assert.deepEqual(again.autter, CARRIER);
});

test("injectCarrier writes only the carrier fields (no attributes, user context or baggage)", () => {
const job = injectCarrier({}, { ...CARRIER, user: { email: "a@example.com" }, attributes: { x: 1 }, baggage: "k=v" });
assert.deepEqual(Object.keys(job.autter).sort(), ["op", "req", "traceparent", "v"]);
const headers = injectCarrier({}, { ...CARRIER, user: "a@example.com" }, "headers");
assert.deepEqual(Object.keys(JSON.parse(headers.autter)).sort(), ["op", "req", "traceparent", "v"]);
});

test("injectCarrier/extractCarrier round-trip HTTP headers (case-insensitive name, string value)", () => {
const headers = { "content-type": "application/json", Autter: "stale" };
const out = injectCarrier(headers, CARRIER, "headers");
assert.deepEqual(headers, { "content-type": "application/json", Autter: "stale" }, "input untouched");
assert.equal(typeof out.autter, "string");
assert.equal(out.Autter, undefined, "a differently-cased old header is replaced, not duplicated");
assert.equal(out["content-type"], "application/json");
assert.deepEqual(extractCarrier(out), CARRIER);
// Received with another case, or as Node's single-entry header list.
assert.deepEqual(extractCarrier({ AUTTER: out.autter }), CARRIER);
assert.deepEqual(extractCarrier({ autter: [out.autter] }), CARRIER);
assert.equal(extractCarrier({ autter: [out.autter, out.autter] }), undefined);
// A fetch Headers instance: a new Headers comes back, the input is untouched.
const fetchHeaders = new Headers({ accept: "text/plain" });
const copy = injectCarrier(fetchHeaders, CARRIER);
assert.ok(copy instanceof Headers);
assert.notEqual(copy, fetchHeaders);
assert.equal(fetchHeaders.has("autter"), false);
assert.equal(copy.get("accept"), "text/plain");
assert.deepEqual(extractCarrier(copy), CARRIER);
assert.deepEqual(extractCarrier(new Headers({ AUTTER: JSON.stringify(CARRIER) })), CARRIER);
});

test("injectCarrier/extractCarrier round-trip SQS MessageAttributes and whole SQS messages", () => {
const attributes = { tenant: { DataType: "String", StringValue: "t1" } };
const out = injectCarrier(attributes, CARRIER, "sqs");
assert.deepEqual(Object.keys(attributes), ["tenant"], "input untouched");
assert.equal(out.tenant, attributes.tenant);
assert.equal(out.autter.DataType, "String");
assert.deepEqual(JSON.parse(out.autter.StringValue), CARRIER);
assert.deepEqual(extractCarrier(out), CARRIER);
// A received message (AWS SDK) and a Lambda SQS event record (lower-case keys).
assert.deepEqual(extractCarrier({ Body: "{}", MessageAttributes: out }), CARRIER);
assert.deepEqual(
extractCarrier({ body: "{}", messageAttributes: { autter: { dataType: "String", stringValue: out.autter.StringValue } } }),
CARRIER,
);
// A non-string attribute type is not a carrier.
assert.equal(extractCarrier({ autter: { DataType: "Binary", StringValue: out.autter.StringValue } }), undefined);
});

test("extractCarrier returns undefined for missing, malformed or oversized input and never throws", () => {
const hostile = new Proxy({}, { get() { throw new Error("boom"); }, has() { throw new Error("boom"); }, ownKeys() { throw new Error("boom"); }, getOwnPropertyDescriptor() { throw new Error("boom"); } });
const malformed = [
undefined, null, 0, 42, true, "", "not json", "[]", [], [CARRIER], {}, { other: 1 },
{ autter: null }, { autter: 7 }, { autter: "not json" }, { autter: { v: 2, op: "x" } },
{ autter: { v: 1, op: "bad op id!" } }, { autter: { v: 1 } }, { v: 1, op: "x".repeat(129) },
{ autter: JSON.stringify({ ...CARRIER, pad: "x".repeat(1100) }) },
{ MessageAttributes: { autter: { DataType: "String", StringValue: "x".repeat(2000) } } },
{ MessageAttributes: { MessageAttributes: { autter: { DataType: "String", StringValue: JSON.stringify(CARRIER) } } } },
{ autter: { StringValue: 5 } },
Object.create({ autter: CARRIER }),
hostile,
{ autter: hostile },
];
malformed.forEach((input, index) => assert.equal(extractCarrier(input), undefined, `malformed input #${index}`));
// Invalid optional fields are dropped, as parseCarrier does.
assert.deepEqual(extractCarrier({ autter: { v: 1, op: "op-1", req: "bad id!", traceparent: "junk" } }), { v: 1, op: "op-1" });
});

test("injectCarrier leaves the target unchanged when there is nothing to inject or nowhere to put it", () => {
const data = { a: 1 };
assert.equal(injectCarrier(data, undefined), data);
assert.equal(injectCarrier(data, null), data);
assert.equal(injectCarrier(data, { v: 2, op: "x" }), data);
assert.deepEqual(data, { a: 1 });
const list = [1, 2];
assert.equal(injectCarrier(list, CARRIER), list);
assert.equal(injectCarrier("text", CARRIER), "text");
assert.equal(injectCarrier(null, CARRIER), null);
class Job { constructor() { this.id = 1; } }
const instance = new Job();
assert.equal(injectCarrier(instance, CARRIER), instance, "class instances are not copied (that would drop their prototype)");
assert.equal(injectCarrier(Object.freeze({ a: 1 }), CARRIER).autter.op, CARRIER.op, "frozen input is fine: it is copied");
});

test("request helpers: ids, route normalisation, ignore globs, outcomes", () => {
assert.equal(resolveRequestId("abc12345"), "abc12345");
assert.match(resolveRequestId("short"), /^[0-9a-f-]{36}$/);
Expand Down
2 changes: 2 additions & 0 deletions packages/runtime-next/src/server-stand-ins.ts
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,8 @@ export function serverStandIns(build: RootBuild): ServerStandIns {
return fn(operation());
},
carrier: () => undefined,
// No operation here, so nothing to carry: the target comes back unchanged.
inject: (target) => target,
},
flushRuntimeLogs: async () => ({ ok: true, errors: [] }),
runtimeLogStats: () => ({ buffered: 0, dropped: 0 }),
Expand Down
Loading
Loading