Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions src/lib/redirects.ts
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@ export const redirectMap: Record<string, string> = {
// Evaluation features/ pages folded into guides/ (revamp); redirect the retired paths
'/docs/evaluation/features/cicd': '/docs/evaluation/guides/cicd',
'/docs/evaluation/features/custom': '/docs/evaluation/guides/custom-evals',
'/docs/evaluation/features/custom-models': '/docs/evaluation/guides/custom-models',
'/docs/evaluation/features/evaluate': '/docs/evaluation/guides/running-evaluations',
'/docs/evaluation/features/futureagi-models': '/docs/evaluation/concepts/evaluator-models',
'/docs/evaluation/concepts/eval-results': '/docs/evaluation/reference/output-types',
Expand Down
4 changes: 3 additions & 1 deletion src/pages/docs/evaluation/guides/custom-models.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,9 @@ With **From model Provider** selected, pick a provider from the **Model Provider
<img src="/images/docs/evaluation/guides/custom-models/provider-dropdown.png" alt="Model Provider dropdown open, listing Open AI, AWS Bedrock, AWS Sagemaker, Vertex AI, and Azure, with a callout pointing at Open AI" style={{ borderRadius: '5px' }} />
*Five supported providers, each with its own credential form*

Fill in the provider's form: a **Model Name** to recognize it later, **Input** and **Output Token Cost Per Million Tokens** for cost tracking, and the provider's credentials, an API key for Open AI, region and access keys for Bedrock or Sagemaker, a service account for Vertex AI, endpoint and key for Azure. A token cost can be 0, for a model you host yourself for example, but not negative. A **Form** and **JSON** toggle lets you fill the fields individually or paste a raw config.
Fill in the provider's form: a **Model Name**, **Input** and **Output Token Cost Per Million Tokens** for cost tracking, and the provider's credentials, an API key for Open AI, region and access keys for Bedrock or Sagemaker, a service account for Vertex AI, endpoint and key for Azure. A token cost can be 0, for a model you host yourself for example, but not negative. A **Form** and **JSON** toggle lets you fill the fields individually or paste a raw config.

For AWS Bedrock, **Model Name is the invocation ID**, not a label you invent. Copy a chat foundation-model ID such as `anthropic.claude-3-haiku-20240307-v1:0`, or a system inference-profile ID such as `us.anthropic.claude-3-5-sonnet-20241022-v2:0`, from the AWS console. A family name such as `Claude` is rejected. Connecting a model does not train or import one in AWS. Save sends a small validation request to AWS and can incur provider charges. New application-profile ARNs, provisioned throughput, and session-token or role credentials are not supported by this form.

<img src="/images/docs/evaluation/guides/custom-models/openai-form.png" alt="Add Model drawer with Open AI selected as the Model Provider, showing Model Name, Input and Output Token Cost Per Million Tokens, a Form/JSON toggle, API Key, and an optional Base URL field, with a callout pointing at the API Key field" style={{ borderRadius: '5px' }} />
*Open AI's form: model name, token costs, API key, and an optional base URL*
Expand Down
Loading