Skip to content

lemonade: a failed backend install during serve leaves the spawned lemond running #729

Open

Description

@rominf

What happens

In the Lemonade engine's serve path (engines/lemonade/src/lib.rs, the router branch after spawn_lemond), two early returns leave the spawned lemond running:

let mut child = crate::process::spawn_lemond(...)?;
...
crate::process::wait_for_lemonade_cli_status(...)
    .context("Lemonade server did not become ready")?;          // returns, child left running
let backend = crate::backend_alignment::ensure_best_llamacpp_backend(...)
    .context("failed to select a supported Lemonade llama.cpp backend")?; // same

The later failure paths in the same function (load_result failing, or the router not reporting the model) do call terminate_pid(child.id(), true) and child.wait() before returning. Its sibling install_best_llamacpp_backend in backend_alignment.rs does the same unconditionally around the same two calls.

Why it matters

When the llama.cpp backend install fails, which is exactly the case rocm serve already retries once and then tells the user to rocm engines install lemonade --reinstall, the engine exits with an error but its lemond keeps running. It still holds the port and its state.

Not verified

Whether anything on the rocm serve side later reaps that lemond, for example a stop through the service record that recorded its pid via write_running_state(..., Some(child.id()), "running"). This was found by reading the code. In the e2e suite the harness's teardown kills it, so no test shows the leak.

Suggested fix

Run both calls inside a closure and terminate the child on its Err, as install_best_llamacpp_backend does. Add a unit or e2e assertion that no lemond survives a failed backend install; @id:serve-lemonade-preparation-recovery already drives that failure with a planted runtime once #351 lands.

Raised in review on #351.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions