diff --git a/README.md b/README.md index 7848990ec..d3c45ec46 100644 --- a/README.md +++ b/README.md @@ -164,6 +164,94 @@ install -m 0755 target/release/rocm target/release/rocmd ~/.local/bin/ See [CONTRIBUTING.md](https://github.com/ROCm/rocm-cli/blob/main/CONTRIBUTING.md) for the full development setup, test commands, and commit-signing requirements. +## Quickstart + +This quickstart takes you from installing ROCm CLI to getting a chat reply from +a model served on your GPU. You need a computer with a supported AMD GPU running +Linux, WSL2, or Windows. The first run involves large downloads: +`rocm install sdk` fetches a ROCm runtime and a PyTorch stack, and `rocm serve` +fetches a model of a few GB. Allow extra time and disk space. + +Follow these steps: + +1. Install ROCm CLI with the command for your platform in + [Installation](#installation). Then open a new terminal and confirm that + `rocm version` prints the CLI version. + +2. Check the GPU, the ROCm runtime, and the serving engine: + + ``` + rocm examine + ``` + + Two lines in the output decide the next steps. For example, on a Radeon or + Ryzen AI GPU: + + ``` + active_runtime_status: ready + effective_default_engine: lemonade + ``` + + On an Instinct GPU, the second line reads `effective_default_engine: vllm` + instead. vLLM isn't available on native Windows, so Windows always uses + `lemonade`. The lines mean: + + - `active_runtime_status`: `ready` means a managed ROCm runtime is already + active, so you can skip step 3. + - `effective_default_engine`: `lemonade` or `vllm`. The engine decides which + model you serve in step 4. + +3. Install a managed ROCm runtime: + + ``` + rocm install sdk + ``` + + This downloads TheRock ROCm wheels and a matching PyTorch stack into a + managed environment. Run `rocm examine` again and confirm that + `active_runtime_status` reports `ready`. + +4. Serve a model in the form your engine accepts: + + - `lemonade` (Ryzen AI and Radeon GPUs) serves GGUF models: + + ``` + rocm serve qwen + ``` + + - `vllm` (Instinct GPUs) serves safetensors models: + + ``` + rocm serve Qwen/Qwen2.5-1.5B-Instruct + ``` + + The server runs in the background. When it's ready, `rocm serve` prints a + deployment summary with the endpoint, the service ID, and the commands that + stop the service and show its logs. + +5. Send the model a prompt. Pass the same model name you served: + + ``` + rocm chat --provider local --model qwen --prompt "Explain ROCm in one sentence" + ``` + + On a `vllm` engine, use `--model Qwen/Qwen2.5-1.5B-Instruct`. To open the + interactive chat instead, run `rocm chat` with no options. Chat is a preview + feature. + +6. Stop the server with the `stop` command from the deployment summary. It has + this form: + + ``` + rocm services stop --yes + ``` + + To find the service ID later, run `rocm services list`. + +For other models and engine options, see +[Configure ROCm and serve a model](#configure-rocm-and-serve-a-model) and +[Model serving](#model-serving). + ## First run Launch ROCm CLI with no arguments: @@ -281,8 +369,12 @@ After the install finishes, serve a model: rocm serve qwen ``` -`qwen` is a built-in alias for a small assistant model that serves out of the -box. You can also serve any compatible Hugging Face model directly. See +`qwen` is a built-in alias for a small GGUF assistant model, served by the +Lemonade engine on Ryzen AI and Radeon GPUs. On Instinct GPUs, which serve +through vLLM, pass a safetensors model instead, such as +`rocm serve Qwen/Qwen2.5-1.5B-Instruct`. + +You can also serve any compatible Hugging Face model directly. See [Model serving](#model-serving) for the GGUF versus safetensors rule, because which form works depends on the engine your GPU selects. diff --git a/docs/rocm-docs/getting-started.md b/docs/rocm-docs/getting-started.md index 46373f7c4..bf3fea56e 100644 --- a/docs/rocm-docs/getting-started.md +++ b/docs/rocm-docs/getting-started.md @@ -6,6 +6,38 @@ SPDX-License-Identifier: MIT # Getting started with ROCm CLI +## Quickstart + + +This quickstart takes you from installing ROCm CLI to getting a chat reply from +a model served on your GPU. You need a computer with a supported AMD GPU running +Linux, WSL2, or Windows. The first run involves large downloads: +`rocm install sdk` fetches a ROCm runtime and a PyTorch stack, and `rocm serve` +fetches a model of a few GB. Allow extra time and disk space. + +Follow these steps: + +1. Install ROCm CLI with the command for your platform in + [Installing ROCm CLI](install/installation.md). Then open a new terminal and + confirm that `rocm version` prints the CLI version. + +```{include} ../../README.md +:start-after: "`rocm version` prints the CLI version." +:end-before: "For other models and engine options, see" +``` + +For other models and engine options, see +[Configure ROCm and serve a model](#configure-rocm-and-serve-a-model) and +[Model serving](commands.md#model-serving). + +The following recording shows the CLI inspecting the environment and chatting +with a locally served model: + +![ROCm CLI demo](https://raw.githubusercontent.com/ROCm/rocm-cli/media/cli.gif) + ## First run ```{include} ../../README.md @@ -13,8 +45,6 @@ SPDX-License-Identifier: MIT :end-before: "## Interactive interfaces" ``` -![ROCm CLI demo](https://raw.githubusercontent.com/ROCm/rocm-cli/media/cli.gif) - ## Interactive interfaces ```{include} ../../README.md diff --git a/docs/rocm-docs/index.md b/docs/rocm-docs/index.md index 203072d63..7c0a9349d 100644 --- a/docs/rocm-docs/index.md +++ b/docs/rocm-docs/index.md @@ -50,7 +50,7 @@ Check platform support and install the prebuilt binaries. :link: getting-started :link-type: doc -Inspect your environment, configure ROCm, and serve a model. +Quickstart: install ROCm CLI, serve a model, and chat with it. ``` ```{grid-item-card} Look up a command