Skip to content

fix(logs): read every log tail through one bounded, lossy routine and stop reporting a log with content as empty - #961

Merged
jussielo-amd merged 18 commits into
mainfrom
split-log-tail-routine
Nov 9, 2026
Merged

jussielo-amd merged 18 commits into
mainfrom
split-log-tail-routine

Conversation

@jussielo-amd

@jussielo-amd jussielo-amd commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Closes ROCMAI-879.

Summary

Three places each read the last lines of a log their own way: the ComfyUI log view (whole file, UTF-16 decode, ANSI strip, lossy text), the rocm services logs / rocm logs text in services_text.rs (a streamed line reader that failed on any invalid text), and the rocmd MCP service_logs tool (whole file into one string, also failing on invalid text). Meanwhile the seek-bounded routine in rocm-core served only the lemonade and vLLM adapters. The three copies disagreed on what a log is, so a stray byte that is not valid text made rocm logs show a single "failed to read" placeholder instead of the readable lines around it, and a log whose newest line is longer than the read budget was reported as empty or as having no output.

This PR reads every one of those tails through one bounded, lossy routine in rocm-core (tail_lines_decoded / tail_log_decoded, taking a LogDecode: PLAIN for UTF-8, CONSOLE for ComfyUI's UTF-16 plus ANSI stripping). The private readers are deleted; each caller keeps a thin adapter that picks a LogDecode, passes the shared LOG_VIEW_MAX_TAIL_READ budget (4 MiB) and adds its own error label. rocmd cannot depend on rocm, which is why the routine lives in rocm-core.

Intended user-visible differences:

  • A log larger than 4 MiB is read from its last 4 MiB, not whole (all three views).
  • Invalid UTF-8 no longer fails the view: rocm services logs, rocm logs and the service_logs tool show the readable text with U+FFFD for the bad bytes instead of a placeholder line or an error. A UTF-16 service log is now shown as lossy text rather than as an error.
  • The tail read never goes past the file size it measured first, so a log that is still being written cannot pull it further, and a trailing fragment made only of escape sequences (a colour reset after the last newline) is not counted as a line.
  • A log that is truncated or recreated between the size being taken and the read (a restarted service recreates its log) no longer fails the whole view with "failed to fill whole buffer": the read measures the log again and starts over from the new size, and a log that will not hold still reads as empty. A log that did not shrink is read exactly as before.
  • A log that has content is never reported as empty or as having no output. When the newest line is longer than the read budget, rocm services logs prints Recent output: none shown; the newest line is longer than the 4 MiB a log view reads, and rocm comfyui logs says the same instead of The saved log is empty.. A genuinely empty log still says no output yet / empty.
  • The service_logs tool (the daemon's MCP server and the CLI's own tool bridge behind rocm mcp-call) now gives three distinct answers through one shared sentence, rocm_core::service_log_tool_summary: no log output yet, log output exists but none is shown because the newest line exceeds the read bound, or how many lines were read. Its structured reply carries newest_line_exceeds_read_bound. The line count is now the number of lines returned, not the number asked for: in the summary and in the structured lines field (asking for 50 lines of a three-line log gives "lines": 3), with the number asked for, after clamping to 1 to 500, in a new lines_requested field. Nothing in the tree reads the field. A missing log file is still an error.
  • ComfyUI logs: ANSI stripping is per line, so an unterminated escape sequence ends at its line instead of swallowing the lines after it.
  • Error wording for a missing log: failed to read <label> <path> with the open step in the cause (services text previously said failed to open <label> <path>).
  • A limit of 0 returns no lines without opening the file in all three readers. Callers clamp the limit, so only direct calls see it.

Split from #929 at the reviewer's request; the other parts are #960 (stop routine), #959 (engine IPC client) and the respawn guard, which follows once those two have merged.

Commits, in reading order

  1. Characterisation tests that pin each reader's behaviour before the change (they pass on the old code).
  2. The convergence onto the shared routine, with the tests updated to the new behaviour plus tests for the bound (a counting reader shows a read touches no more than its budget) and every budget from nothing to the whole file for UTF-8 and UTF-16.
  3. Scenario logs-08 (invalid byte).
  4. The UTF-8 size-snapshot and trailing-escape-fragment fix.
  5. The "log with content is not empty" fix and scenarios logs-09 to logs-12.
  6. The service_logs tool's empty-versus-over-long answer and scenarios logs-13 and logs-14.
  7. A doc-comment style fix, then the docs/testing.md paragraph for the scenarios.
  8. A log whose only line exceeds the read bound is no longer shown as absent by the logs browser, the lifecycle summary or the engine startup failure detail, with scenario logs-15.
  9. The tail read measures a log again when it shrinks between the size and the read, with tests over a source that shrinks at each read point.
  10. The private follow cursor in services_text.rs is renamed FollowCursor, so it no longer shares a name with rocm_core::LogTail; the architecture entry says which is which. A rename only.
  11. A clippy fix in the new test fixture.
  12. The structured lines field of the service_logs tool reports the lines returned, with lines_requested beside it, in both copies of the tool, with unit tests and scenario logs-16.
  13. The UTF-16 detection doc comment now says half of the odd bytes (the code, which is the rule the ComfyUI reader used, is unchanged), with a boundary test.
  14. docs/manual-testing.md describes the new messages and the tool's fields.

Test plan

Ran locally (debug build, one target dir, targeted rather than the full workspace gate):

  • cargo test -p rocm-core --lib log_tail: 40 passed (including the shrinking-source tests and the UTF-16 detection boundary test); the whole rocm-core crate passed at 1273 tests in the previous round, before that boundary test was added.
  • cargo test -p rocmd: 204 passed, 2 ignored (includes service_logs_tell_an_empty_log_from_one_whose_newest_line_exceeds_the_read_bound and service_logs_report_the_lines_returned_and_the_lines_asked_for_apart).
  • cargo test -p rocm --bin rocm whole suite with --no-fail-fast: 1712 passed, 9 ignored, 0 failed (also the filtered services_text, comfyui:: and chat:: groups).
  • The logs scenarios through the real harness with a debug rocm/rocmd built with --features rocm/e2e-test-hooks and -n 'logs-': 16 scenarios, 47 steps, all passed; feature_naming and cargo test -p xtask prepush pass.
  • cargo fmt --all --check, cargo clippy --locked -p rocm-core -p rocmd -p rocm --all-targets -- -D warnings, cargo clippy --locked -p e2e-cucumber --test e2e -- -D warnings, cargo xtask manifest --check, cargo xtask check-crate-edges, cargo xtask check-architecture-doc, cargo xtask lychee: all clean.
  • Leak scan of origin/main..HEAD read through: the only hits are identifiers such as run_internal_mcp_call already in the tree.
  • Mutation checks: making the lossy decode drop U+FFFD turns read_tail_lines_decodes_invalid_eight_bit_text_lossily and the matching rocm-core tests red; making newest_line_exceeds_the_read_bound ignore the skipped-byte count turns an_empty_tail_says_whether_the_log_is_empty_or_its_newest_line_is_too_long, the_service_log_summary_names_each_of_the_three_states_apart, service_logs_say_no_output_only_for_an_empty_log_and_name_the_bound_otherwise and the rocmd service_logs test red. Both mutations were reverted. Removing the shrink handling (the probe's early-end case, or the two short-window checks) turns three of the shrinking-source tests red each; the same tests fail with failed to fill whole buffer or an empty tail before the fix. Reporting the requested count in lines turns the unit test of each copy of the service_logs tool and scenario logs-16 red, and changing the UTF-16 detection threshold turns the boundary test red.

Not run: the full workspace test gate, scripts/smoke_local.py, cargo xtask e2e in release mode, the coverage floors, and a Windows-target clippy (no platform-gated code is touched). CI is the authoritative run for those.

Scenarios

New scenarios in tests/e2e-cucumber/features/logs.feature, all on the mock lane that runs on every PR (they read planted log files only, no GPU, no network): logs-08 (rocm logs --search over a log with an invalid byte), logs-09 and logs-10 (rocm services logs for an over-long line and an empty log), logs-11 and logs-12 (rocm comfyui logs for the same pair), logs-13 and logs-14 (rocm mcp-call service_logs for the same pair). logs-15 (rocm logs over a command log whose only line exceeds the bound, which prints the omitted-output line instead of No logs found yet.) and logs-16 (rocm mcp-call service_logs for 50 lines of a three-line log: lines is 3 and lines_requested is 50). The daemon's copy of the service_logs tool is covered by a rocmd unit test, because the harness drives only the rocm binary. Existing scenarios logs-01 to logs-07 are unchanged.

Three modules read the last lines of a log their own way: the ComfyUI log
view (whole file, sixteen-bit decode, ANSI strip, lossy text), the services
text module (streamed, fails on invalid text) and the rocmd MCP service_logs
tool (whole file into one string, fails on invalid text), while the
seek-bounded routine in rocm-core serves only the engines.

Add characterisation tests that pass on the unmodified code and document the
differences before they are removed: a large file (the whole-file readers
return a line far outside any tail window; the shared routine does not), a
sixteen-bit little-endian log with ANSI colour codes, invalid eight-bit text,
a missing file, a limit of zero, fewer lines than the limit, a final line
without a trailing newline, and CRLF endings. Test-only; no behaviour change.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
The ComfyUI log view, the services text module and the rocmd MCP
service_logs tool each read a log tail their own way, while the seek-bounded
rocm_core::tail_lines served only the engines. Converge all of them on it.

rocm_core::log_tail gains tail_lines_decoded, taking a LogDecode: PLAIN
(UTF-8, what the engines and the service logs use) or CONSOLE (UTF-16
little-endian detected by byte order mark or zero-byte pattern, ANSI
sequences stripped; what the ComfyUI log view used). The UTF-16 path keeps
the seek-boundary handling: the window starts on a character boundary and a
partial first line is dropped. strip_ansi_sequences is exported for the
assistant-prose note in tool_policy. A shared LOG_VIEW_MAX_TAIL_READ budget
(4 MiB) is what the three log views pass. The private readers are deleted;
each caller keeps a thin adapter that picks a LogDecode and its own error
label.

User-visible differences:
- A log larger than 4 MiB is read from its last 4 MiB, not whole (all three).
- Invalid UTF-8 no longer fails the view: `rocm services logs` and
  `rocm logs` showed one "<failed to read ...>" placeholder line and the MCP
  service_logs tool returned an error; both now show the readable text with
  U+FFFD for the bad bytes.
- Service logs and the MCP tool still read UTF-8 and keep ANSI sequences; a
  UTF-16 service log is now shown as lossy text rather than as an error.
- ComfyUI logs: ANSI stripping is per line, so an unterminated escape
  sequence ends at its line instead of swallowing the lines after it.
- Error wording: a missing log is "failed to read <label> <path>" (cause:
  "failed to open <path>") in services text, was "failed to open <label>
  <path>"; ComfyUI and MCP name the open step in the cause.
- A limit of 0 returns no lines without opening the file in all three (the
  MCP reader used to return the whole file; ComfyUI and MCP used to fail on
  a missing file first). Callers clamp the limit, so only direct calls see it.

The characterisation tests from the previous commit are updated to the new
behaviour, and new tests cover the bound (a counting reader shows a read
touches no more than its budget), every budget from nothing to the whole file
for UTF-8 and UTF-16, and each caller through the shared routine.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
`rocm logs --search` reads the CLI's action logs through the shared log-tail
routine. Add scenario logs-04: an action log with one byte that is not valid
text still yields every matching line. Before the log views were converged
on the shared routine, that byte made the whole file collapse into a single
"failed to read" placeholder line and the search found nothing.

The planting step gains an option for the stray byte; the existing steps and
the match-count assertion are reused.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
… trailing escape fragment

Two ways the shared tail read disagreed with what it advertises, both found in review.

The UTF-8 branch seeked from the size taken by the earlier stat and then read to the current end of the file, so a log whose writer kept appending was consumed past the byte budget by however much it had grown. The sixteen-bit branch already capped its read with take(size - window_start); the UTF-8 branch now does the same. The new test hands the routine a source that grows on its first read and asserts the lines stop where the snapshot ended and the bytes touched stay within the window; it is red with the cap removed.

A console log that ends in a colour reset after its last newline ended in an unterminated fragment made only of an escape sequence. The routine counted it as a line, so with a limit of one it evicted the real last line and returned an empty string, where the ComfyUI reader it replaced (strip ANSI from the whole text, then split) returned the line. A final unterminated fragment that strips to nothing is no longer counted when ANSI stripping is on; a terminated blank line, including one made only of an escape sequence, is still a line, and plain decoding strips nothing so it keeps the fragment. The tests cover UTF-8 and UTF-16 with and without a byte order mark, and compare the result with a copy of the old reader on inputs where the two agree.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…utput

The log views read at most the last 4 MiB of a log and drop a partial first line. When the newest line is itself longer than that bound the window holds only a fragment of it, so the read returned no lines and the views printed "The saved log is empty." (rocm comfyui logs) and "Recent output: no output yet" (rocm services logs) for a file that is not empty.

The byte bound stays. rocm-core gains tail_log_decoded, which returns the same lines in a LogTail that also records how many bytes at the start of the file the budget left unread, and LogTail::newest_line_exceeds_the_read_bound tells an empty tail of a log with content from an empty log (or a limit of zero). The two views use it: an empty file still says it is empty, and a log whose newest line does not fit says so and names the bound, derived from LOG_VIEW_MAX_TAIL_READ so the number cannot drift from it.

The unit tests and four new scenarios (logs-05 to logs-08, one pair per view) assert each message together with the log on disk that it describes: the size and the absence of a newline for the long line, zero bytes for the empty file. The rocmd MCP service_logs tool and the chat and launch-failure readers print no claim about emptiness, so they are unchanged.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…s the read bound in the service_logs tool

The tool read a log's tail without saying what the byte budget left unread, so an empty log and a log whose newest line is longer than the 4 MiB a log view reads both came back as "Read the last 80 line(s)" with an empty tail. The CLI log views already separate the two; the tool is the same class. It now answers through one shared sentence, rocm_core::service_log_tool_summary (no output yet, newest line too long to show, or how many lines were read), and its structured reply carries newest_line_exceeds_read_bound. Both copies of the tool use it: the daemon's MCP server and the CLI's own tool bridge (rocm mcp-call). The notice clause moves from the CLI into rocm-core so the daemon, which cannot depend on the CLI, words it the same way, derived from the read budget.

The line count in the summary is now the number of lines returned, not the number asked for. A missing log file is still an error.

Scenarios logs-09 and logs-10 run rocm mcp-call service_logs against an over-long line and an empty log and check the reply against the file on disk; the daemon's copy is covered by a rocmd unit test, since the harness drives only the rocm binary.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…hort

clippy's too_long_first_doc_paragraph rejects the two doc comments added with the service_logs summary; split each into a one-sentence summary and the rest.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…utine

The log views now read a log tail through one bounded, lossy routine and the
logs scenarios cover it, but docs/testing.md only described the service_logs
tool. Say which scenario covers which view (logs-08 to logs-12) and where the
routine's own tests run, next to the existing service_logs paragraph.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
@jussielo-amd
jussielo-amd marked this pull request as ready for review November 9, 2026 07:08
@jussielo-amd
jussielo-amd requested a review from a team as a code owner November 9, 2026 07:08
@jussielo-amd
jussielo-amd requested review from siloteemu and a balanced review from Copilot November 9, 2026 07:08

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The lines-only adapter discards oversized-line metadata, causing misleading empty-log or no-activity messages instead of an omitted-output placeholder.

1 open issue
What changed in this PR

This PR consolidates log-tail reading in rocm-core so the CLI, ComfyUI view, and daemon share bounded, lossy decoding without introducing a dependency between the two applications.

Changes:

  • Replaces separate readers with shared tail routines and a 4 MiB log-view budget.
  • Distinguishes empty logs from oversized lines and standardizes service_logs summaries.
  • Adds decoding, read-bound, and output regression tests, plus seven end-to-end scenarios.
File Description
tests/​e2e-cucumber/​tests/​e2e/​service_cleanup_steps.rs Exposes record fixtures for log scenarios.
tests/​e2e-cucumber/​tests/​e2e/​logs_steps.rs Adds log fixtures and CLI/tool output assertions.
tests/​e2e-cucumber/​features/​logs.feature Adds invalid-byte, oversized-line, and empty-log scenarios.
docs/​testing.md Documents log-tail behavior and regression coverage.
docs/​architecture.md Updates shared-reader and caller descriptions.
crates/​rocm-core/​src/​log_tail.rs Implements shared decoding, bounded reads, summaries, and tests.
crates/​rocm-core/​src/​lib.rs Exports shared log-tail APIs.
apps/​rocmd/​src/​mcp.rs Uses shared tails and summaries for service_logs.
apps/​rocm/​src/​tool_policy.rs Updates a comment naming the ANSI-stripping helper.
apps/​rocm/​src/​services_text.rs Adopts bounded reads and oversized-line notices.
apps/​rocm/​src/​comfyui.rs Replaces private decoding with the shared console reader.
apps/​rocm/​src/​chat/​mcp_bridge.rs Aligns CLI tool replies with shared log summaries.

Review effort: Balanced

Comment thread apps/rocm/src/services_text.rs Resolved
…e logs browser

The lines-only reader behind the logs browser dropped what the byte budget left unread, so a command, screen or lifecycle log whose only line is longer than the 4 MiB a log view reads came back as an empty list. `rocm logs` then printed "No logs found yet." and the recent activity summary said "no activity yet" although the files are on disk. The same reader feeds the managed-engine startup failure detail, which likewise left the log out.

read_optional_tail_lines now returns one placeholder line, `<no output shown from <label>: the newest line is longer than the 4 MiB a log view reads>`, for such a log, the way it already returns one `<failed to read ...>` line for a read error; a missing or empty log is still an empty list. The recent activity summary and the lifecycle entries of the browser read the tail with its bound and say "none shown" with the same clause the service log view uses, instead of formatting a placeholder as a lifecycle record.

Unit tests cover the reader, the browser for a command and a screen log, the lifecycle summary and the startup failure detail, and scenario logs-15 runs `rocm logs` against a command log whose only line exceeds the bound and checks the printed line together with the file on disk.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are recorded for the shared log-tail changes.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@r0x0r r0x0r left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went through this one carefully and it holds up, so I'm approving it. Two nits left inline, neither of them a blocker.

The thing I most wanted to break was the extracted tail routine, so I ran it differentially against a whole-file reference: 268,286 combinations of content, budget and decoding, covering UTF-16 both with and without a BOM, CRLF, a lone 0xff, seeks landing mid-multibyte, and ANSI. Zero mismatches. The only way I could provoke a false newest_line_exceeds was with a budget of two bytes or less, which the 4 MiB callers can never reach.

The tests are the kind I like to see: each one asserts the message together with the on-disk state that message describes, per AGENTS.md section 3. I also walked the PR description against the diff claim for claim, and it matches.

Of the two inline notes, the truncation race on the probe read is the one I'd act on before merge. It isn't a regression, tail_lines has had it all along, but this PR widens its reach from the engine adapters to every log view and the service_logs tool. The other one is a naming collision that will cost somebody an afternoon later on.

CI is green apart from E2E tests (W7900, current), which is still queued; every other E2E lane passed.

Comment thread crates/rocm-core/src/log_tail.rs Outdated
Comment on lines 203 to 210
source
.seek(SeekFrom::Start(seek_pos - unit))
.with_context(|| format!("failed to seek in {}", path.display()))?;
let mut probe = [0u8; 1];
file.read_exact(&mut probe)
let mut probe = [0u8; 2];
let probe = &mut probe[..unit as usize];
source
.read_exact(probe)
.with_context(|| format!("failed to read from {}", path.display()))?;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A log truncated between the stat and this probe read now fails the whole view. read_exact on the probe byte returns UnexpectedEof if the file shrank below seek_pos after metadata.len() was taken. I ran the production routine against a shrunk source to confirm: Err("failed to read from t: failed to fill whole buffer").

The race is real rather than theoretical. service_launch.rs:891 does fs::File::create(&record.log_path) on every restart, and services_text.rs's own follow cursor already handles exactly this case ("Truncated or replaced (a restart reuses the path)").

It is pre-existing in tail_lines, but this PR extends its reach from the engine adapters to every log view and the service_logs tool, reached via apps/rocm/src/services_text.rs:1377, apps/rocmd/src/mcp.rs:850 and read_log_tail in apps/rocm/src/comfyui.rs:683.

Fix: treat a short probe or window read as "the file shrank" and fall back to window_start = 0.

This is the one I'd act on before merge, though to be clear it is not a regression relative to today's behaviour in the adapters.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and fixed in 8840d0e. The routine took the size once and then probed the byte before its window with read_exact, so a log truncated between the two failed the whole view with failed to fill whole buffer; the race is real because a restart recreates the service log. I reproduced it by running the production routine over a source that shrinks when a read reaches a chosen offset, before changing the code. There was a second shrink point beyond the one the comment names: a log that shrank after the probe made the window read come back short, which showed no lines at all rather than an error.

The read now treats an early end of the source at the probe, and a window (UTF-8 or UTF-16) shorter than the size it was given, as the log having shrunk. It measures the source again and starts over from that size, which reads a log that is now small enough from its start and reads one that is still larger than the budget from its new end. A log that keeps shrinking through three passes reads as empty rather than failing. A log that did not shrink takes the same path as before. The callers all reach this one function (the engine adapters through tail_lines, services_text.rs, the rocmd service_logs tool and the ComfyUI view), so none needed a change; their suites pass.

Tests in log_tail.rs use a pure ShrinkingSource, so they run on every platform: a_log_that_shrank_before_the_probe_is_measured_again_instead_of_failing, a_log_that_shrank_before_the_window_read_is_measured_again, a_log_that_shrank_but_is_still_larger_than_the_budget_is_read_from_its_new_end, a_sixteen_bit_log_that_shrank_is_measured_again_at_the_probe_and_the_window, a_whole_file_read_of_a_log_that_shrank_is_measured_again, a_log_that_keeps_shrinking_ends_in_an_empty_tail_not_an_error and a_log_that_did_not_shrink_reads_exactly_as_before. With the probe's early-end case disabled, or with the short-window checks disabled, three of them fail each time.

Comment thread docs/architecture.md Outdated
- [`service_records.rs`](../apps/rocm/src/service_records.rs) — mechanical relocation, owns types: removing one managed-service record and every file it owns (`remove_managed_service_record`, `service_record_artifacts`) and the `rocm services prune` plan/render/apply steps (`build_service_prune_plan`, `render_service_prune_plan`, `apply_service_prune_plan`, `prune_managed_service_records`) with the age gate behind them (`DEFAULT_SERVICE_PRUNE_MIN_AGE_HOURS`, `service_prune_min_age_hours`). It reads record state through [`managed_service_status.rs`](../apps/rocm/src/managed_service_status.rs) and [`services_text.rs`](../apps/rocm/src/services_text.rs) and builds its advice text with [`service_hints.rs`](../apps/rocm/src/service_hints.rs); none of the three depends on it. `remove_managed_service_record`, `prune_managed_service_records`, `service_prune_min_age_hours` and `DEFAULT_SERVICE_PRUNE_MIN_AGE_HOURS` are `pub(crate)` (`services()` in [`services_cmd.rs`](../apps/rocm/src/services_cmd.rs) calls the first three; the `ServicesCommand::Prune` clap definition in [`cli.rs`](../apps/rocm/src/cli.rs) uses the constant), and `ServicePruneOutcome`, the return type of `prune_managed_service_records`, is `pub(crate)` with `pub(crate)` `text`, `removed_records`, `removed_files` and `failures` fields that `services()` reads. Every other item is private. The cluster's regression tests live in this module's own `mod tests`, reaching the shared `test_paths`, `plant_service_record` and `plant_service_record_without_its_key` fixtures through `crate::tests::` (the last two widened to `pub(crate)` for that).
- [`services_cmd.rs`](../apps/rocm/src/services_cmd.rs) — mechanical relocation, dispatch-adjacent, no owned types: `rocm services`' command handler (`services()`, which dispatches `list`, `logs`, `stop`, `restart`, `remove` and `prune`) and `run_approved_service_action`, the approval-gated stop/restart path. `ServicesCommand` lives in [`cli.rs`](../apps/rocm/src/cli.rs). Both functions are `pub(crate)`: `dispatch()` calls `services()` through a re-import, and [`advised_commands.rs`](../apps/rocm/src/advised_commands.rs)'s test reaches `run_approved_service_action` through a test-only re-import in `main.rs`. The module reads from [`services_text.rs`](../apps/rocm/src/services_text.rs), [`service_control.rs`](../apps/rocm/src/service_control.rs), [`service_records.rs`](../apps/rocm/src/service_records.rs), [`service_hints.rs`](../apps/rocm/src/service_hints.rs) and [`cli_audit.rs`](../apps/rocm/src/cli_audit.rs); none of them calls back into it. The cluster's one regression test lives in this module's own `mod tests`, reaching the shared `test_paths` fixture through `crate::tests::`.
- [`services_text.rs`](../apps/rocm/src/services_text.rs) — mechanical relocation, owns private types (Phase 6f, ROCMAI-209): the CLI/TUI logs browser (`render_logs_text`, `render_logs_browser_text`, `render_logs_browser_page_text`, `render_logs_browser_page_text_with_options`, `logs_browser_total_pages`, the `LogBrowserEntry` type, `collect_log_browser_entries`, `filter_log_browser_entries`, `log_browser_source_label`, `write_past_attempts_hint`), the `rocm services`/`rocm services logs` status surfaces (`render_services_text`, `render_services_json`, `render_service_logs_text`, `render_service_logs_text_with_options`, `render_service_action_result`, `read_managed_service_manifest`, `load_managed_service`), the `--follow` tail behind `rocm logs --service <id> --follow` and `rocm services logs <id> --follow` (`follow_service_logs`, `follow_service_logs_with`, the `LogTail`, `LogSegment` and `FollowFooter` types, `write_log_segments`, `newly_ready`; new code written in this module, not relocated), the internal `rocmd`-compatible status surface (`render_internal_status_text`), the `stop_server` answer shape and its renderer (`internal_stop_server_value`, `service_stop_unconfirmed`, `render_unconfirmed_service_stop`, `process_check_command`), plus `validate_service_id`, `list_log_files`, `read_optional_tail_lines`, and the CLI-lifecycle-log tail formatting helpers (`format_cli_lifecycle_tail_line`, `lifecycle_event_label`, `humanize_log_label`, `lifecycle_field`, `read_tail_lines`). The managed-service stop and restart implementation that used to live here is in [`service_control.rs`](../apps/rocm/src/service_control.rs). The `LogBrowserEntry` struct moved here from `main.rs` with the logs browser and stays private. `dispatch()` calls directly into this module at several points — `render_internal_status_text`, `render_service_logs_text`, `render_logs_browser_text`, and `render_logs_text` — all re-imported via `use crate::services_text::{...};`; [`services_cmd.rs`](../apps/rocm/src/services_cmd.rs)'s `services()` calls `render_services_text`, `render_services_json` and `render_service_logs_text` through its own imports. `render_service_action_result`, `service_stop_unconfirmed` and `validate_service_id` are also `pub(crate)` and read from `services_cmd.rs`'s `run_approved_service_action`; `validate_service_id` is also read from [`service_records.rs`](../apps/rocm/src/service_records.rs)'s `remove_managed_service_record`/`service_record_artifacts`, and `load_managed_service` from its `remove_managed_service_record`/`apply_service_prune_plan`. `load_managed_service` is `pub(crate)`, read from [`runtime_services.rs`](../apps/rocm/src/runtime_services.rs) (reconciling live servers against a runtime activation), from [`service_control.rs`](../apps/rocm/src/service_control.rs) and from [`managed_service_status.rs`](../apps/rocm/src/managed_service_status.rs)'s own test module; `read_managed_service_manifest` and `internal_stop_server_value` are `pub(crate)` for `service_control.rs`, and `process_check_command` for its tests; `read_tail_lines`, `render_logs_text`, `render_logs_browser_text`, `render_services_text`, `render_service_logs_text`, and `validate_service_id` are also `pub(crate)`, read from [`chat/mcp_bridge.rs`](../apps/rocm/src/chat/mcp_bridge.rs)'s read-only command runner; `read_optional_tail_lines` and `render_service_action_result` are also `pub(crate)`, read from [`service_launch.rs`](../apps/rocm/src/service_launch.rs)'s managed-spawn/attach path. Every other item in this list has no caller outside this module and stays private. The cluster's regression tests live in this module's own `mod tests`, reaching the shared `test_paths` fixture through `crate::tests::` the same way other mechanically relocated modules do; `assert_no_retry_line` stays here as a `pub(crate)` test helper because it checks the unconfirmed-stop rendering and `service_launch.rs`'s tests read it too. Two tests that call this module's `load_managed_service` only incidentally — their primary subject is `runtime_services::pin_service_record_to_runtime`/`restore_service_record_pin` and `runtime_services::restart_service_onto_runtime_with`'s outcome classification respectively — stay in `main.rs`'s shared `mod tests` instead.
- [`services_text.rs`](../apps/rocm/src/services_text.rs) — mechanical relocation, owns private types (Phase 6f, ROCMAI-209): the CLI/TUI logs browser (`render_logs_text`, `render_logs_browser_text`, `render_logs_browser_page_text`, `render_logs_browser_page_text_with_options`, `logs_browser_total_pages`, the `LogBrowserEntry` type, `collect_log_browser_entries`, `filter_log_browser_entries`, `log_browser_source_label`, `write_past_attempts_hint`), the `rocm services`/`rocm services logs` status surfaces (`render_services_text`, `render_services_json`, `render_service_logs_text`, `render_service_logs_text_with_options`, `render_service_action_result`, `read_managed_service_manifest`, `load_managed_service`), the `--follow` tail behind `rocm logs --service <id> --follow` and `rocm services logs <id> --follow` (`follow_service_logs`, `follow_service_logs_with`, the `LogTail`, `LogSegment` and `FollowFooter` types, `write_log_segments`, `newly_ready`; new code written in this module, not relocated), the internal `rocmd`-compatible status surface (`render_internal_status_text`), the `stop_server` answer shape and its renderer (`internal_stop_server_value`, `service_stop_unconfirmed`, `render_unconfirmed_service_stop`, `process_check_command`), plus `validate_service_id`, `list_log_files`, `read_optional_tail_lines`, and the CLI-lifecycle-log tail formatting helpers (`format_cli_lifecycle_tail_line`, `lifecycle_event_label`, `humanize_log_label`, `lifecycle_field`, `read_log_tail`). The managed-service stop and restart implementation that used to live here is in [`service_control.rs`](../apps/rocm/src/service_control.rs). The `LogBrowserEntry` struct moved here from `main.rs` with the logs browser and stays private. `dispatch()` calls directly into this module at several points — `render_internal_status_text`, `render_service_logs_text`, `render_logs_browser_text`, and `render_logs_text` — all re-imported via `use crate::services_text::{...};`; [`services_cmd.rs`](../apps/rocm/src/services_cmd.rs)'s `services()` calls `render_services_text`, `render_services_json` and `render_service_logs_text` through its own imports. `render_service_action_result`, `service_stop_unconfirmed` and `validate_service_id` are also `pub(crate)` and read from `services_cmd.rs`'s `run_approved_service_action`; `validate_service_id` is also read from [`service_records.rs`](../apps/rocm/src/service_records.rs)'s `remove_managed_service_record`/`service_record_artifacts`, and `load_managed_service` from its `remove_managed_service_record`/`apply_service_prune_plan`. `load_managed_service` is `pub(crate)`, read from [`runtime_services.rs`](../apps/rocm/src/runtime_services.rs) (reconciling live servers against a runtime activation), from [`service_control.rs`](../apps/rocm/src/service_control.rs) and from [`managed_service_status.rs`](../apps/rocm/src/managed_service_status.rs)'s own test module; `read_managed_service_manifest` and `internal_stop_server_value` are `pub(crate)` for `service_control.rs`, and `process_check_command` for its tests; `read_log_tail`, `render_logs_text`, `render_logs_browser_text`, `render_services_text`, `render_service_logs_text`, and `validate_service_id` are also `pub(crate)`, read from [`chat/mcp_bridge.rs`](../apps/rocm/src/chat/mcp_bridge.rs)'s read-only command runner; `read_optional_tail_lines` and `render_service_action_result` are also `pub(crate)`, read from [`service_launch.rs`](../apps/rocm/src/service_launch.rs)'s managed-spawn/attach path. Every other item in this list has no caller outside this module and stays private. The cluster's regression tests live in this module's own `mod tests`, reaching the shared `test_paths` fixture through `crate::tests::` the same way other mechanically relocated modules do; `assert_no_retry_line` stays here as a `pub(crate)` test helper because it checks the unconfirmed-stop rendering and `service_launch.rs`'s tests read it too. Two tests that call this module's `load_managed_service` only incidentally — their primary subject is `runtime_services::pin_service_record_to_runtime`/`restore_service_record_pin` and `runtime_services::restart_service_onto_runtime_with`'s outcome classification respectively — stay in `main.rs`'s shared `mod tests` instead.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Anchored on this doc line because it's the half of the issue that lands in the diff; the code half is apps/rocm/src/services_text.rs:564.

Two different LogTail types now live in services_text.rs: the private follow cursor declared at services_text.rs:564, and rocm_core::LogTail, the bounded-read result, used throughout the same file. This entry still lists "the LogTail ... type" without distinguishing them. AGENTS.md section 6 names this exact hazard for the two "spinner" types.

Renaming the follow cursor, to FollowCursor say, would settle both.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, and renamed in aaa6670. The private follow cursor in services_text.rs is now FollowCursor, with its constructor calls, tests and comments, and its doc comment, the module comment and the architecture entry say that it follows a growing file and is not rocm_core::LogTail, the result of a bounded tail read, which the entry now links. It is a rename only. I also swept the rest of this PR for same-named types: LogDecode and LogTail exist only in rocm-core, the new private Pass in log_tail.rs has no namesake in that crate, and the three read_log_tail adapters are functions that each wrap the one routine for their own caller, not types.

…he read

The tail read took the file size once and then probed the byte before its window with read_exact. A service restart recreates its log, so a log truncated between the size and that probe made read_exact fail with "failed to fill whole buffer" and the whole view with it; a log that shrank after the probe read short and showed no lines.

The read now treats an early end of the source, at the probe or in the window (UTF-8 and UTF-16), as the log having shrunk: it measures the source again and starts over from that size, which reads a log that is now small enough from its start. After a few passes a log that will not hold still reads as empty rather than failing. A log that did not shrink is read exactly as before.

Tests run the production routine over a source that shrinks when a read reaches a chosen offset: before the probe, before the window, still larger than the budget, UTF-16 at both points, a whole-file read, and a source that never settles.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…sult

services_text.rs declared a private LogTail, the cursor that reads what a growing log gained for --follow, and also uses rocm_core::LogTail, the result of a bounded tail read, so one file held two unrelated types of the same name and the architecture entry listed "the LogTail type" without saying which. Rename the private one to FollowCursor, with its constructor calls, tests and comments, say in its doc comment and in the architecture entry which is which, and link the shared one. A rename only; no behaviour changes.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…terator

Clippy rejects collecting format! results into a String; write the lines with writeln! instead. Test-only.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the shared, bounded log-tail reader changes.

No open issues 🎉

Review effort: Balanced

@jussielo-amd
jussielo-amd added this pull request to the merge queue Nov 9, 2026

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head ab0695bb91a

Outcome: needs work — 3 blocking findings, all narrow and cheaply fixable; CI is fully green and the new bounded/lossy tail routine itself (crates/rocm-core/src/log_tail.rs) held up well under a dedicated review pass (boundary math, the shrink/retry race handling, and the trailing-escape-fragment logic all checked out against the code and its tests).

Blocking

  • MCP service_logs structured reply echoes the requested line limit, not the lines actually returned — apps/rocmd/src/mcp.rs:637, apps/rocm/src/chat/mcp_bridge.rs:221
    Both call sites build json!({ ..., "lines": lines, ..., "tail": log_tail.lines.join("\n") }) where lines is the clamped requested limit (arguments.get("lines")...clamp(1, 500)), while tail and the prose summary (rocm_core::service_log_tool_summary) correctly use log_tail.lines.len() — the actual count returned, per the PR's own stated fix. So requesting 50 lines on a 10-line log yields "lines": 50 next to a 10-line tail: the prose and structured halves of the same reply disagree, and nothing in the test suite asserts the structured field's value.
    Confidence 85 · logic · Fix: "lines": log_tail.lines.len() at both sites (or rename the field if echoing the request is deliberately wanted, and add a separate "returned" count).

  • UTF-16 sniff doc comment misstates its own threshold by 2x — crates/rocm-core/src/log_tail.rs:33-34 (doc) vs :418 (code)
    The doc says UTF-16-without-BOM is detected "when at least a quarter of the odd-numbered bytes in its first 512 bytes are zero." The code checks zero_odd_bytes * 4 >= head.len(); since odd-indexed bytes are ~half of head.len(), this actually requires half of the odd bytes to be zero, not a quarter. Low functional risk (genuine mostly-ASCII UTF-16 text is close to 100% zero odd bytes either way), but it's a new, factually wrong claim about new public-API behavior.
    Confidence 85 · mechanical · Fix: say "at least half" instead of "a quarter" (or loosen the code to zero_odd_bytes * 8 >= head.len() if a quarter was the intent).

  • docs/manual-testing.md not updated for the new observable messages, despite the repo's own contributor guide requiring it in the same change — docs/manual-testing.md (untouched; see the existing rocm services logs walkthrough)
    This PR introduces new user-visible strings (e.g. "Recent output: none shown; the newest line is longer than the 4 MiB a log view reads", the ComfyUI equivalent, and the MCP tool's three-way summary) and updates docs/testing.md and docs/architecture.md accordingly, but the manual-testing walkthrough for rocm services logs still doesn't mention the new empty-vs-oversized-line distinction.
    Confidence 80 · mechanical · Fix: add a short bullet to the manual-testing walkthrough covering the new distinction, or state explicitly in the PR why it's intentionally left to the e2e scenarios.

Non-blocking / decisions for the author

None beyond the above.


What this covered: full diff 3dc4a31a…ab0695bb (12 files, ~2.6k added lines) — the shared log_tail.rs routine, all three callers (services_text.rs, comfyui.rs, apps/rocmd/src/mcp.rs, chat/mcp_bridge.rs), the new e2e scenarios/steps, and docs — each read in full by an independent pass, plus history/comment/agent-instruction checks. CI on the PR is green across the board.

Not a human review. This is an automated code review; please verify findings independently before acting on them.

@jussielo-amd
jussielo-amd removed this pull request from the merge queue due to a manual request Nov 9, 2026
…mber asked for

The structured reply put the requested line limit in "lines" while "tail" and the summary used the lines actually read, so asking for 50 lines of a ten-line log said "lines": 50 beside a ten-line tail. "lines" is now the number of lines returned, and the number asked for (after clamping to 1 to 500) is reported as "lines_requested". Both copies of the tool change: the daemon's MCP server and the CLI's own tool bridge behind rocm mcp-call. Nothing in the tree reads the field.

A unit test per copy asks for more lines than the log has, for fewer, and for an empty log, and checks both fields; scenario logs-16 runs rocm mcp-call service_logs for 50 lines of a three-line log and checks the reply against the file on disk.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
LogDecode documented the detection of a log without a byte order mark as a quarter of the odd-numbered bytes in its first 512 bytes being zero, but the check is zero_odd_bytes * 4 >= sampled length, which is half of the odd positions. The check is the rule the ComfyUI reader used before the views shared one routine, so the documentation changes rather than the rule. A test pins the boundary: 4 of 8 odd bytes zero is UTF-16 and 3 is not, likewise 128 and 127 within the 512 byte sample, and a source under 8 bytes is never UTF-16.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…om one they cannot show

Add the observable messages to the local server records walkthrough: the none-shown line for a log whose newest line is longer than the 4 MiB a log view reads in rocm services logs, rocm logs and rocm comfyui logs, the lossy decoding of a stray byte, and the three summaries and the lines and lines_requested fields of the service_logs tool.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The manual test places the over-long line in a server log that rocm logs does not scan, so it cannot produce the expected browser placeholder.

1 open issue

Review effort: Balanced

Comment thread docs/manual-testing.md Resolved
The walkthrough told the reader to replace a server's log file and then expected rocm logs to list it, but without --service that command reads the activity log and the files under logs/cli/ and logs/tui/ in the data folder, never a server's log. Split the check by view and give each the exact line it prints: the server log through rocm services logs or rocm logs --service, the rocm logs browser over a file in logs/cli/ (and the activity log), and rocm comfyui logs over a file under apps/comfyui/logs/. The service_logs tool bullet now says it reads the same server log.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported for the shared log-tail consolidation.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@jussielo-amd
jussielo-amd added this pull request to the merge queue Nov 9, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 9, 2026
Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…n Windows

PR 966 re-exports managed_engine_startup_failure_detail from service_launch only off Windows, but the services_text log-tail tests call it through crate::service_launch and are compiled on Windows too, so the Windows clippy leg failed with E0425 in the merge queue. Re-export it for Windows test builds only: non-test Windows builds gain no unused import and Linux is unchanged.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Caution

3 potential secret finding(s) were committed in this pull request. Treat each credential as compromised and revoke or rotate it now, even if a later commit removed it. Deleting the line or force-pushing does not undo the leak: the commits stay reachable from this pull request and from every clone or fork, and automated scanners harvest secrets pushed to public repositories within minutes.

If a finding is a test fixture or a false positive, allowlist it in the configuration of the scanner that reported it.

Security scan found 3 critical or high finding(s) on lines changed by this pull request
Scanner Severity Rule Location Finding
Gitleaks HIGH generic-api-key apps/rocm/src/runtime_install_finalize.rs:1315 Detected a Generic API Key, potentially exposing access to various services and sensitive operations.
Gitleaks HIGH generic-api-key apps/rocm/src/serve_summary.rs:521 Detected a Generic API Key, potentially exposing access to various services and sensitive operations.
Gitleaks HIGH generic-api-key apps/rocm/src/serve_summary.rs:549 Detected a Generic API Key, potentially exposing access to various services and sensitive operations.

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

@jussielo-amd
jussielo-amd added this pull request to the merge queue Nov 9, 2026
Merged via the queue into main with commit bf8c334 Nov 9, 2026
38 of 41 checks passed
@jussielo-amd
jussielo-amd deleted the split-log-tail-routine branch November 9, 2026 22:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants