data [required]
\n[object]
List of custom evaluator configuration data objects.
attributes [required]
\nobject
Attributes of a custom Agent Observability evaluator configuration.
category
\nstring
Category of the evaluator.
created_at [required]
\ndate-time
Timestamp when the evaluator configuration was created.
created_by
\nobject
A Datadog user associated with a custom evaluator configuration.
string
Email address of the user.
eval_name [required]
\nstring
Name of the custom evaluator.
last_updated_by
\nobject
A Datadog user associated with a custom evaluator configuration.
string
Email address of the user.
llm_judge_config
\nobject
LLM judge configuration for a custom evaluator.
assessment_criteria
\nobject
Criteria used to assess the pass/fail result of a custom evaluator.
max_threshold
\ndouble
Maximum numeric threshold for a passing result.
min_threshold
\ndouble
Minimum numeric threshold for a passing result.
pass_values
\n[string]
Specific output values considered as a passing result.
pass_when
\nboolean
When true, a boolean output of true is treated as passing.
context_query
\nstring
Query used to extract additional context for the evaluation.
inference_params [required]
\nobject
LLM inference parameters for a custom evaluator.
frequency_penalty
\ndouble
Frequency penalty to reduce repetition.
max_tokens
\nint64
Maximum number of tokens to generate.
presence_penalty
\ndouble
Presence penalty to reduce repetition.
temperature
\ndouble
Sampling temperature for the LLM.
top_k
\nint64
Top-k sampling parameter.
top_p
\ndouble
Top-p (nucleus) sampling parameter.
last_used_library_prompt_template_name
\nstring
Name of the last library prompt template used.
modified_library_prompt_template
\nboolean
Whether the library prompt template was modified.
output_schema
\nobject
JSON schema describing the expected output format of the LLM judge.
parsing_type
\nenum
Output parsing type for a custom LLM judge evaluator. \nAllowed enum values: structured_output,json,keyword_search
prompt_template
\n[object]
List of messages forming the LLM judge prompt template.
content
\nstring
Text content of the message.
contents
\n[object]
Multi-part content blocks for the message.
type [required]
\nstring
Content block type.
value [required]
\nobject
Value of a prompt message content block.
text
\nstring
Text content of the message block.
tool_call
\nobject
A tool call within a prompt message.
arguments
\nstring
JSON-encoded arguments for the tool call.
id
\nstring
Unique identifier of the tool call.
name
\nstring
Name of the tool being called.
type
\nstring
Type of the tool call.
tool_call_result
\nobject
A tool call result within a prompt message.
name
\nstring
Name of the tool that produced this result.
result
\nstring
The result returned by the tool.
tool_id
\nstring
Identifier of the tool call this result corresponds to.
type
\nstring
Type of the tool result.
role [required]
\nstring
Role of the message author.
target_query
\nstring
Query used to extract the target value to evaluate.
user_specified_json_post_processing_function
\nstring
User-provided function applied to post-process the JSON output of the LLM judge.
llm_provider
\nobject
LLM provider configuration for a custom evaluator.
bedrock
\nobject
AWS Bedrock-specific options for LLM provider configuration.
inference_profile
\nstring
Bedrock inference profile identifier, such as an application inference profile ARN.
region
\nstring
AWS region for Bedrock.
integration_account_id
\nstring
Integration account identifier.
integration_provider
\nenum
Name of the LLM integration provider. \nAllowed enum values: openai,amazon-bedrock,anthropic,azure-openai,vertex-ai,llm-proxy
model_name
\nstring
Name of the LLM model.
vertex_ai
\nobject
Google Vertex AI-specific options for LLM provider configuration.
location
\nstring
Google Cloud region.
project
\nstring
Google Cloud project ID.
target
\nobject
Target application configuration for a custom evaluator.
application_name [required]
\nstring
Name of the ML application this evaluator targets.
enabled [required]
\nboolean
Whether the evaluator is active for the target application.
eval_scope
\nenum
Scope at which to evaluate spans. \nAllowed enum values: span,trace,session
experiment_project_ids
\n[string]
Experiment project IDs this evaluator is scoped to.
filter
\nstring
Filter expression to select which spans to evaluate.
root_spans_only
\nboolean
When true, only root spans are evaluated.
sampling_percentage
\ndouble
Percentage of traces to evaluate. Must be greater than 0 and at most 100.
updated_at [required]
\ndate-time
Timestamp when the evaluator configuration was last updated.
id [required]
\nstring
Unique name identifier of the evaluator configuration.
type [required]
\nenum
Type of the custom Agent Observability evaluator configuration resource. \nAllowed enum values: evaluator_config
data [required]
\n[object]
List of custom evaluator configuration data objects.
attributes [required]
\nobject
Attributes of a custom Agent Observability evaluator configuration.
category
\nstring
Category of the evaluator.
created_at [required]
\ndate-time
Timestamp when the evaluator configuration was created.
created_by
\nobject
A Datadog user associated with a custom evaluator configuration.
string
Email address of the user.
eval_name [required]
\nstring
Name of the custom evaluator.
last_updated_by
\nobject
A Datadog user associated with a custom evaluator configuration.
string
Email address of the user.
llm_judge_config
\nobject
LLM judge configuration for a custom evaluator.
assessment_criteria
\nobject
Criteria used to assess the pass/fail result of a custom evaluator.
max_threshold
\ndouble
Maximum numeric threshold for a passing result.
min_threshold
\ndouble
Minimum numeric threshold for a passing result.
pass_values
\n[string]
Specific output values considered as a passing result.
pass_when
\nboolean
When true, a boolean output of true is treated as passing.
context_query
\nstring
Query used to extract additional context for the evaluation.
inference_params [required]
\nobject
LLM inference parameters for a custom evaluator.
frequency_penalty
\ndouble
Frequency penalty to reduce repetition.
max_tokens
\nint64
Maximum number of tokens to generate.
presence_penalty
\ndouble
Presence penalty to reduce repetition.
temperature
\ndouble
Sampling temperature for the LLM.
top_k
\nint64
Top-k sampling parameter.
top_p
\ndouble
Top-p (nucleus) sampling parameter.
last_used_library_prompt_template_name
\nstring
Name of the last library prompt template used.
modified_library_prompt_template
\nboolean
Whether the library prompt template was modified.
output_schema
\nobject
JSON schema describing the expected output format of the LLM judge. Omit for a deterministic evaluator.
parsing_type
\nenum
Output parsing type for a custom LLM judge evaluator. \nAllowed enum values: structured_output,json,keyword_search,typed_questions
prompt_template
\n[object]
List of messages forming the LLM judge prompt template. Omit for a deterministic evaluator.
content
\nstring
Text content of the message.
contents
\n[object]
Multi-part content blocks for the message.
type [required]
\nstring
Content block type.
value [required]
\nobject
Value of a prompt message content block.
text
\nstring
Text content of the message block.
tool_call
\nobject
A tool call within a prompt message.
arguments
\nstring
JSON-encoded arguments for the tool call.
id
\nstring
Unique identifier of the tool call.
name
\nstring
Name of the tool being called.
type
\nstring
Type of the tool call.
tool_call_result
\nobject
A tool call result within a prompt message.
name
\nstring
Name of the tool that produced this result.
result
\nstring
The result returned by the tool.
tool_id
\nstring
Identifier of the tool call this result corresponds to.
type
\nstring
Type of the tool result.
role [required]
\nstring
Role of the message author.
target_query
\nstring
Query used to extract the target value to evaluate.
user_specified_json_post_processing_function
\nstring
User-provided JavaScript function applied to post-process the JSON output of the LLM judge. For a deterministic evaluator it is the evaluator itself: it receives the evaluated span (span scope), {spans} (trace scope), or {session_id, traces} (session scope), and its return value sets the result. A boolean, number, string or object becomes a boolean, score, categorical or JSON metric; null skips the evaluation; and {value, assessment, reasoning, metadata} also sets pass or fail, reasoning, and metadata.
llm_provider
\nobject
LLM provider configuration for a custom evaluator.
bedrock
\nobject
AWS Bedrock-specific options for LLM provider configuration.
inference_profile
\nstring
Bedrock inference profile identifier, such as an application inference profile ARN.
region
\nstring
AWS region for Bedrock.
integration_account_id
\nstring
Integration account identifier.
integration_provider
\nenum
Name of the LLM integration provider. Use deterministic for a code evaluator, which runs llm_judge_config.user_specified_json_post_processing_function instead of calling an LLM. insights_sponsored is returned for evaluators that Datadog manages and cannot be set. \nAllowed enum values: openai,amazon_bedrock,anthropic,azure_openai,vertex_ai,llmproxy,deterministic,typesafe,datadog,datadog_ai_gatewayShow 1 more
,insights_sponsored
model_name
\nstring
Name of the LLM model. Use javascript for a deterministic evaluator.
vertex_ai
\nobject
Google Vertex AI-specific options for LLM provider configuration.
location
\nstring
Google Cloud region.
project
\nstring
Google Cloud project ID.
target
\nobject
Target application configuration for a custom evaluator.
application_name [required]
\nstring
Name of the ML application this evaluator targets.
enabled [required]
\nboolean
Whether the evaluator is active for the target application.
eval_scope
\nenum
Scope at which to evaluate spans. trace_fan_out requires llm_judge_config.target_query. \nAllowed enum values: span,trace,trace_fan_out,session
experiment_project_ids
\n[string]
Experiment project IDs this evaluator is scoped to.
filter
\nstring
Filter expression to select which spans to evaluate.
root_spans_only
\nboolean
When true, only root spans are evaluated.
sampling_percentage
\ndouble
Percentage of traces to evaluate. Must be greater than 0 and at most 100.
updated_at [required]
\ndate-time
Timestamp when the evaluator configuration was last updated.
id [required]
\nstring
Unique name identifier of the evaluator configuration.
type [required]
\nenum
Type of the custom Agent Observability evaluator configuration resource. \nAllowed enum values: evaluator_config
data [required]
\nobject
Data object for a custom Agent Observability evaluator configuration.
attributes [required]
\nobject
Attributes of a custom Agent Observability evaluator configuration.
category
\nstring
Category of the evaluator.
created_at [required]
\ndate-time
Timestamp when the evaluator configuration was created.
created_by
\nobject
A Datadog user associated with a custom evaluator configuration.
string
Email address of the user.
eval_name [required]
\nstring
Name of the custom evaluator.
last_updated_by
\nobject
A Datadog user associated with a custom evaluator configuration.
string
Email address of the user.
llm_judge_config
\nobject
LLM judge configuration for a custom evaluator.
assessment_criteria
\nobject
Criteria used to assess the pass/fail result of a custom evaluator.
max_threshold
\ndouble
Maximum numeric threshold for a passing result.
min_threshold
\ndouble
Minimum numeric threshold for a passing result.
pass_values
\n[string]
Specific output values considered as a passing result.
pass_when
\nboolean
When true, a boolean output of true is treated as passing.
context_query
\nstring
Query used to extract additional context for the evaluation.
inference_params [required]
\nobject
LLM inference parameters for a custom evaluator.
frequency_penalty
\ndouble
Frequency penalty to reduce repetition.
max_tokens
\nint64
Maximum number of tokens to generate.
presence_penalty
\ndouble
Presence penalty to reduce repetition.
temperature
\ndouble
Sampling temperature for the LLM.
top_k
\nint64
Top-k sampling parameter.
top_p
\ndouble
Top-p (nucleus) sampling parameter.
last_used_library_prompt_template_name
\nstring
Name of the last library prompt template used.
modified_library_prompt_template
\nboolean
Whether the library prompt template was modified.
output_schema
\nobject
JSON schema describing the expected output format of the LLM judge.
parsing_type
\nenum
Output parsing type for a custom LLM judge evaluator. \nAllowed enum values: structured_output,json,keyword_search
prompt_template
\n[object]
List of messages forming the LLM judge prompt template.
content
\nstring
Text content of the message.
contents
\n[object]
Multi-part content blocks for the message.
type [required]
\nstring
Content block type.
value [required]
\nobject
Value of a prompt message content block.
text
\nstring
Text content of the message block.
tool_call
\nobject
A tool call within a prompt message.
arguments
\nstring
JSON-encoded arguments for the tool call.
id
\nstring
Unique identifier of the tool call.
name
\nstring
Name of the tool being called.
type
\nstring
Type of the tool call.
tool_call_result
\nobject
A tool call result within a prompt message.
name
\nstring
Name of the tool that produced this result.
result
\nstring
The result returned by the tool.
tool_id
\nstring
Identifier of the tool call this result corresponds to.
type
\nstring
Type of the tool result.
role [required]
\nstring
Role of the message author.
target_query
\nstring
Query used to extract the target value to evaluate.
user_specified_json_post_processing_function
\nstring
User-provided function applied to post-process the JSON output of the LLM judge.
llm_provider
\nobject
LLM provider configuration for a custom evaluator.
bedrock
\nobject
AWS Bedrock-specific options for LLM provider configuration.
inference_profile
\nstring
Bedrock inference profile identifier, such as an application inference profile ARN.
region
\nstring
AWS region for Bedrock.
integration_account_id
\nstring
Integration account identifier.
integration_provider
\nenum
Name of the LLM integration provider. \nAllowed enum values: openai,amazon-bedrock,anthropic,azure-openai,vertex-ai,llm-proxy
model_name
\nstring
Name of the LLM model.
vertex_ai
\nobject
Google Vertex AI-specific options for LLM provider configuration.
location
\nstring
Google Cloud region.
project
\nstring
Google Cloud project ID.
target
\nobject
Target application configuration for a custom evaluator.
application_name [required]
\nstring
Name of the ML application this evaluator targets.
enabled [required]
\nboolean
Whether the evaluator is active for the target application.
eval_scope
\nenum
Scope at which to evaluate spans. \nAllowed enum values: span,trace,session
experiment_project_ids
\n[string]
Experiment project IDs this evaluator is scoped to.
filter
\nstring
Filter expression to select which spans to evaluate.
root_spans_only
\nboolean
When true, only root spans are evaluated.
sampling_percentage
\ndouble
Percentage of traces to evaluate. Must be greater than 0 and at most 100.
updated_at [required]
\ndate-time
Timestamp when the evaluator configuration was last updated.
id [required]
\nstring
Unique name identifier of the evaluator configuration.
type [required]
\nenum
Type of the custom Agent Observability evaluator configuration resource. \nAllowed enum values: evaluator_config
data [required]
\nobject
Data object for a custom Agent Observability evaluator configuration.
attributes [required]
\nobject
Attributes of a custom Agent Observability evaluator configuration.
category
\nstring
Category of the evaluator.
created_at [required]
\ndate-time
Timestamp when the evaluator configuration was created.
created_by
\nobject
A Datadog user associated with a custom evaluator configuration.
string
Email address of the user.
eval_name [required]
\nstring
Name of the custom evaluator.
last_updated_by
\nobject
A Datadog user associated with a custom evaluator configuration.
string
Email address of the user.
llm_judge_config
\nobject
LLM judge configuration for a custom evaluator.
assessment_criteria
\nobject
Criteria used to assess the pass/fail result of a custom evaluator.
max_threshold
\ndouble
Maximum numeric threshold for a passing result.
min_threshold
\ndouble
Minimum numeric threshold for a passing result.
pass_values
\n[string]
Specific output values considered as a passing result.
pass_when
\nboolean
When true, a boolean output of true is treated as passing.
context_query
\nstring
Query used to extract additional context for the evaluation.
inference_params [required]
\nobject
LLM inference parameters for a custom evaluator.
frequency_penalty
\ndouble
Frequency penalty to reduce repetition.
max_tokens
\nint64
Maximum number of tokens to generate.
presence_penalty
\ndouble
Presence penalty to reduce repetition.
temperature
\ndouble
Sampling temperature for the LLM.
top_k
\nint64
Top-k sampling parameter.
top_p
\ndouble
Top-p (nucleus) sampling parameter.
last_used_library_prompt_template_name
\nstring
Name of the last library prompt template used.
modified_library_prompt_template
\nboolean
Whether the library prompt template was modified.
output_schema
\nobject
JSON schema describing the expected output format of the LLM judge. Omit for a deterministic evaluator.
parsing_type
\nenum
Output parsing type for a custom LLM judge evaluator. \nAllowed enum values: structured_output,json,keyword_search,typed_questions
prompt_template
\n[object]
List of messages forming the LLM judge prompt template. Omit for a deterministic evaluator.
content
\nstring
Text content of the message.
contents
\n[object]
Multi-part content blocks for the message.
type [required]
\nstring
Content block type.
value [required]
\nobject
Value of a prompt message content block.
text
\nstring
Text content of the message block.
tool_call
\nobject
A tool call within a prompt message.
arguments
\nstring
JSON-encoded arguments for the tool call.
id
\nstring
Unique identifier of the tool call.
name
\nstring
Name of the tool being called.
type
\nstring
Type of the tool call.
tool_call_result
\nobject
A tool call result within a prompt message.
name
\nstring
Name of the tool that produced this result.
result
\nstring
The result returned by the tool.
tool_id
\nstring
Identifier of the tool call this result corresponds to.
type
\nstring
Type of the tool result.
role [required]
\nstring
Role of the message author.
target_query
\nstring
Query used to extract the target value to evaluate.
user_specified_json_post_processing_function
\nstring
User-provided JavaScript function applied to post-process the JSON output of the LLM judge. For a deterministic evaluator it is the evaluator itself: it receives the evaluated span (span scope), {spans} (trace scope), or {session_id, traces} (session scope), and its return value sets the result. A boolean, number, string or object becomes a boolean, score, categorical or JSON metric; null skips the evaluation; and {value, assessment, reasoning, metadata} also sets pass or fail, reasoning, and metadata.
llm_provider
\nobject
LLM provider configuration for a custom evaluator.
bedrock
\nobject
AWS Bedrock-specific options for LLM provider configuration.
inference_profile
\nstring
Bedrock inference profile identifier, such as an application inference profile ARN.
region
\nstring
AWS region for Bedrock.
integration_account_id
\nstring
Integration account identifier.
integration_provider
\nenum
Name of the LLM integration provider. Use deterministic for a code evaluator, which runs llm_judge_config.user_specified_json_post_processing_function instead of calling an LLM. insights_sponsored is returned for evaluators that Datadog manages and cannot be set. \nAllowed enum values: openai,amazon_bedrock,anthropic,azure_openai,vertex_ai,llmproxy,deterministic,typesafe,datadog,datadog_ai_gatewayShow 1 more
,insights_sponsored
model_name
\nstring
Name of the LLM model. Use javascript for a deterministic evaluator.
vertex_ai
\nobject
Google Vertex AI-specific options for LLM provider configuration.
location
\nstring
Google Cloud region.
project
\nstring
Google Cloud project ID.
target
\nobject
Target application configuration for a custom evaluator.
application_name [required]
\nstring
Name of the ML application this evaluator targets.
enabled [required]
\nboolean
Whether the evaluator is active for the target application.
eval_scope
\nenum
Scope at which to evaluate spans. trace_fan_out requires llm_judge_config.target_query. \nAllowed enum values: span,trace,trace_fan_out,session
experiment_project_ids
\n[string]
Experiment project IDs this evaluator is scoped to.
filter
\nstring
Filter expression to select which spans to evaluate.
root_spans_only
\nboolean
When true, only root spans are evaluated.
sampling_percentage
\ndouble
Percentage of traces to evaluate. Must be greater than 0 and at most 100.
updated_at [required]
\ndate-time
Timestamp when the evaluator configuration was last updated.
id [required]
\nstring
Unique name identifier of the evaluator configuration.
type [required]
\nenum
Type of the custom Agent Observability evaluator configuration resource. \nAllowed enum values: evaluator_config
data [required]
\nobject
Data object for creating or updating a custom Agent Observability evaluator configuration.
attributes [required]
\nobject
Attributes for creating or updating a custom Agent Observability evaluator configuration.
category
\nstring
Category of the evaluator.
eval_name
\nstring
Name of the custom evaluator. If provided, must match the eval_name path parameter.
llm_judge_config
\nobject
LLM judge configuration for a custom evaluator.
assessment_criteria
\nobject
Criteria used to assess the pass/fail result of a custom evaluator.
max_threshold
\ndouble
Maximum numeric threshold for a passing result.
min_threshold
\ndouble
Minimum numeric threshold for a passing result.
pass_values
\n[string]
Specific output values considered as a passing result.
pass_when
\nboolean
When true, a boolean output of true is treated as passing.
context_query
\nstring
Query used to extract additional context for the evaluation.
inference_params [required]
\nobject
LLM inference parameters for a custom evaluator.
frequency_penalty
\ndouble
Frequency penalty to reduce repetition.
max_tokens
\nint64
Maximum number of tokens to generate.
presence_penalty
\ndouble
Presence penalty to reduce repetition.
temperature
\ndouble
Sampling temperature for the LLM.
top_k
\nint64
Top-k sampling parameter.
top_p
\ndouble
Top-p (nucleus) sampling parameter.
last_used_library_prompt_template_name
\nstring
Name of the last library prompt template used.
modified_library_prompt_template
\nboolean
Whether the library prompt template was modified.
output_schema
\nobject
JSON schema describing the expected output format of the LLM judge.
parsing_type
\nenum
Output parsing type for a custom LLM judge evaluator. \nAllowed enum values: structured_output,json,keyword_search
prompt_template
\n[object]
List of messages forming the LLM judge prompt template.
content
\nstring
Text content of the message.
contents
\n[object]
Multi-part content blocks for the message.
type [required]
\nstring
Content block type.
value [required]
\nobject
Value of a prompt message content block.
text
\nstring
Text content of the message block.
tool_call
\nobject
A tool call within a prompt message.
arguments
\nstring
JSON-encoded arguments for the tool call.
id
\nstring
Unique identifier of the tool call.
name
\nstring
Name of the tool being called.
type
\nstring
Type of the tool call.
tool_call_result
\nobject
A tool call result within a prompt message.
name
\nstring
Name of the tool that produced this result.
result
\nstring
The result returned by the tool.
tool_id
\nstring
Identifier of the tool call this result corresponds to.
type
\nstring
Type of the tool result.
role [required]
\nstring
Role of the message author.
target_query
\nstring
Query used to extract the target value to evaluate.
user_specified_json_post_processing_function
\nstring
User-provided function applied to post-process the JSON output of the LLM judge.
llm_provider
\nobject
LLM provider configuration for a custom evaluator.
bedrock
\nobject
AWS Bedrock-specific options for LLM provider configuration.
inference_profile
\nstring
Bedrock inference profile identifier, such as an application inference profile ARN.
region
\nstring
AWS region for Bedrock.
integration_account_id
\nstring
Integration account identifier.
integration_provider
\nenum
Name of the LLM integration provider. \nAllowed enum values: openai,amazon-bedrock,anthropic,azure-openai,vertex-ai,llm-proxy
model_name
\nstring
Name of the LLM model.
vertex_ai
\nobject
Google Vertex AI-specific options for LLM provider configuration.
location
\nstring
Google Cloud region.
project
\nstring
Google Cloud project ID.
target [required]
\nobject
Target application configuration for a custom evaluator.
application_name [required]
\nstring
Name of the ML application this evaluator targets.
enabled [required]
\nboolean
Whether the evaluator is active for the target application.
eval_scope
\nenum
Scope at which to evaluate spans. \nAllowed enum values: span,trace,session
experiment_project_ids
\n[string]
Experiment project IDs this evaluator is scoped to.
filter
\nstring
Filter expression to select which spans to evaluate.
root_spans_only
\nboolean
When true, only root spans are evaluated.
sampling_percentage
\ndouble
Percentage of traces to evaluate. Must be greater than 0 and at most 100.
id
\nstring
Name of the evaluator. If provided, must match the eval_name path parameter.
type [required]
\nenum
Type of the custom Agent Observability evaluator configuration resource. \nAllowed enum values: evaluator_config
data [required]
\nobject
Data object for creating or updating a custom Agent Observability evaluator configuration.
attributes [required]
\nobject
Attributes for creating or updating a custom Agent Observability evaluator configuration.
category
\nstring
Category of the evaluator.
eval_name
\nstring
Name of the custom evaluator. If provided, must match the eval_name path parameter.
llm_judge_config
\nobject
LLM judge configuration for a custom evaluator.
assessment_criteria
\nobject
Criteria used to assess the pass/fail result of a custom evaluator.
max_threshold
\ndouble
Maximum numeric threshold for a passing result.
min_threshold
\ndouble
Minimum numeric threshold for a passing result.
pass_values
\n[string]
Specific output values considered as a passing result.
pass_when
\nboolean
When true, a boolean output of true is treated as passing.
context_query
\nstring
Query used to extract additional context for the evaluation.
inference_params [required]
\nobject
LLM inference parameters for a custom evaluator.
frequency_penalty
\ndouble
Frequency penalty to reduce repetition.
max_tokens
\nint64
Maximum number of tokens to generate.
presence_penalty
\ndouble
Presence penalty to reduce repetition.
temperature
\ndouble
Sampling temperature for the LLM.
top_k
\nint64
Top-k sampling parameter.
top_p
\ndouble
Top-p (nucleus) sampling parameter.
last_used_library_prompt_template_name
\nstring
Name of the last library prompt template used.
modified_library_prompt_template
\nboolean
Whether the library prompt template was modified.
output_schema
\nobject
JSON schema describing the expected output format of the LLM judge. Omit for a deterministic evaluator.
parsing_type
\nenum
Output parsing type for a custom LLM judge evaluator. \nAllowed enum values: structured_output,json,keyword_search,typed_questions
prompt_template
\n[object]
List of messages forming the LLM judge prompt template. Omit for a deterministic evaluator.
content
\nstring
Text content of the message.
contents
\n[object]
Multi-part content blocks for the message.
type [required]
\nstring
Content block type.
value [required]
\nobject
Value of a prompt message content block.
text
\nstring
Text content of the message block.
tool_call
\nobject
A tool call within a prompt message.
arguments
\nstring
JSON-encoded arguments for the tool call.
id
\nstring
Unique identifier of the tool call.
name
\nstring
Name of the tool being called.
type
\nstring
Type of the tool call.
tool_call_result
\nobject
A tool call result within a prompt message.
name
\nstring
Name of the tool that produced this result.
result
\nstring
The result returned by the tool.
tool_id
\nstring
Identifier of the tool call this result corresponds to.
type
\nstring
Type of the tool result.
role [required]
\nstring
Role of the message author.
target_query
\nstring
Query used to extract the target value to evaluate.
user_specified_json_post_processing_function
\nstring
User-provided JavaScript function applied to post-process the JSON output of the LLM judge. For a deterministic evaluator it is the evaluator itself: it receives the evaluated span (span scope), {spans} (trace scope), or {session_id, traces} (session scope), and its return value sets the result. A boolean, number, string or object becomes a boolean, score, categorical or JSON metric; null skips the evaluation; and {value, assessment, reasoning, metadata} also sets pass or fail, reasoning, and metadata.
llm_provider
\nobject
LLM provider configuration for a custom evaluator.
bedrock
\nobject
AWS Bedrock-specific options for LLM provider configuration.
inference_profile
\nstring
Bedrock inference profile identifier, such as an application inference profile ARN.
region
\nstring
AWS region for Bedrock.
integration_account_id
\nstring
Integration account identifier.
integration_provider
\nenum
Name of the LLM integration provider. Use deterministic for a code evaluator, which runs llm_judge_config.user_specified_json_post_processing_function instead of calling an LLM. insights_sponsored is returned for evaluators that Datadog manages and cannot be set. \nAllowed enum values: openai,amazon_bedrock,anthropic,azure_openai,vertex_ai,llmproxy,deterministic,typesafe,datadog,datadog_ai_gatewayShow 1 more
,insights_sponsored
model_name
\nstring
Name of the LLM model. Use javascript for a deterministic evaluator.
vertex_ai
\nobject
Google Vertex AI-specific options for LLM provider configuration.
location
\nstring
Google Cloud region.
project
\nstring
Google Cloud project ID.
target [required]
\nobject
Target application configuration for a custom evaluator.
application_name [required]
\nstring
Name of the ML application this evaluator targets.
enabled [required]
\nboolean
Whether the evaluator is active for the target application.
eval_scope
\nenum
Scope at which to evaluate spans. trace_fan_out requires llm_judge_config.target_query. \nAllowed enum values: span,trace,trace_fan_out,session
experiment_project_ids
\n[string]
Experiment project IDs this evaluator is scoped to.
filter
\nstring
Filter expression to select which spans to evaluate.
root_spans_only
\nboolean
When true, only root spans are evaluated.
sampling_percentage
\ndouble
Percentage of traces to evaluate. Must be greater than 0 and at most 100.
id
\nstring
Name of the evaluator. If provided, must match the eval_name path parameter.
type [required]
\nenum
Type of the custom Agent Observability evaluator configuration resource. \nAllowed enum values: evaluator_config