Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 47 additions & 21 deletions backend/app/api/routes/llm.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@

from fastapi import APIRouter, Depends, HTTPException
from opentelemetry import trace
from pydantic import TypeAdapter
from sqlmodel import Session

from app.api.deps import AuthContextDep, SessionDep
from app.api.permissions import Permission, require_permission
Expand All @@ -19,13 +21,50 @@
LLMJobPublic,
JobStatus,
)
from app.models.llm.response import LLMResponse, Usage
from app.models.llm.response import LLMOutput, LLMResponse, Usage
from app.services.llm.jobs import start_job
from app.utils import APIResponse, validate_callback_url, load_description

logger = logging.getLogger(__name__)

router = APIRouter(tags=["LLM"])

_LLM_OUTPUT_ADAPTER: TypeAdapter[LLMOutput] = TypeAdapter(LLMOutput)


def _resolve_llm_output(
raw_content: dict,
project_id: int,
session: Session,
job_id: UUID,
) -> LLMOutput | None:
Comment on lines +35 to +40

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Narrow the helper contract.

dict is unconstrained, and this function never returns None: validation either returns LLMOutput or raises. As per coding guidelines, every parameter and return value needs a narrow type.

Proposed fix
 def _resolve_llm_output(
-    raw_content: dict,
+    raw_content: dict[str, object],
     project_id: int,
     session: Session,
     job_id: UUID,
-) -> LLMOutput | None:
+) -> LLMOutput:
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
def _resolve_llm_output(
raw_content: dict,
project_id: int,
session: Session,
job_id: UUID,
) -> LLMOutput | None:
def _resolve_llm_output(
raw_content: dict[str, object],
project_id: int,
session: Session,
job_id: UUID,
) -> LLMOutput:
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@backend/app/api/routes/llm.py` around lines 35 - 40, Update
_resolve_llm_output to use the narrowest concrete type for raw_content based on
the validated LLM response schema, replacing unconstrained dict, and change its
return annotation from LLMOutput | None to LLMOutput. Preserve the existing
validation behavior where valid input returns LLMOutput and invalid input
raises.

Source: Coding guidelines

"""Parse the persisted `llm_call.content` dict into the typed LLMOutput,
presigning the audio URL in place first.

Persisted TTS content marks a not-yet-presigned S3 path with format="uri" —
not a valid AudioContent literal ("base64"/"url") — so that sentinel must be
resolved to a real "url" before the dict can validate into the typed model.
"""
inner = raw_content.get("content")
if (
raw_content.get("type") == "audio"
and isinstance(inner, dict)
and inner.get("format") == "uri"
):
s3_path = inner.get("value", "")
try:
storage = get_cloud_storage(session, project_id)
inner["value"] = storage.get_signed_url(s3_path, expires_in=3600)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Name the presigned-URL TTL.

3600 is an undocumented policy value. Extract it into a clearly unit-suffixed constant. As per coding guidelines, do not use magic values.

Proposed fix
 _LLM_OUTPUT_ADAPTER: TypeAdapter[LLMOutput] = TypeAdapter(LLMOutput)
+PRESIGNED_AUDIO_URL_TTL_SECONDS = 3_600

-            inner["value"] = storage.get_signed_url(s3_path, expires_in=3600)
+            inner["value"] = storage.get_signed_url(
+                s3_path, expires_in=PRESIGNED_AUDIO_URL_TTL_SECONDS
+            )
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
inner["value"] = storage.get_signed_url(s3_path, expires_in=3600)
_PRESIGNED_AUDIO_URL_TTL_SECONDS = 3_600
inner["value"] = storage.get_signed_url(
s3_path, expires_in=PRESIGNED_AUDIO_URL_TTL_SECONDS
)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@backend/app/api/routes/llm.py` at line 57, Extract the 3600-second expiration
value used in the storage.get_signed_url call within the surrounding route logic
into a clearly named, unit-suffixed constant, then pass that constant as
expires_in. Keep the existing one-hour TTL behavior unchanged.

Source: Coding guidelines

except Exception as e:
logger.warning(
f"[get_llm_call_status] Failed to generate presigned URL for audio: {e} | job_id={job_id}"
)
Comment on lines +59 to +61

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Use the helper’s name in this log prefix.

This warning originates in _resolve_llm_output, not get_llm_call_status; the current prefix misattributes presigning failures. As per coding guidelines, every log line must be prefixed with its function name.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@backend/app/api/routes/llm.py` around lines 59 - 61, Update the
logger.warning call in _resolve_llm_output so its prefix uses
_resolve_llm_output instead of get_llm_call_status, while preserving the
existing error details and job_id context.

Source: Coding guidelines

inner["value"] = ""
inner["format"] = "url"

return _LLM_OUTPUT_ADAPTER.validate_python(raw_content)


llm_callback_router = APIRouter()


Expand Down Expand Up @@ -155,32 +194,19 @@ def get_llm_call_status(
# Get the first LLM call from the list which will be the only call for the job id
# since we initially won't be using this endpoint for llm chains
llm_call = llm_calls[0]
output_payload = copy.deepcopy(llm_call.content)
if (
isinstance(output_payload, dict)
and output_payload.get("type") == "audio"
and isinstance(output_payload.get("content"), dict)
and output_payload["content"].get("format") == "uri"
):
s3_path = output_payload["content"].get("value", "")
try:
storage = get_cloud_storage(session, project_id)
output_payload["content"]["value"] = storage.get_signed_url(
s3_path, expires_in=3600
)
except Exception as e:
logger.warning(
f"[get_llm_call_status] Failed to generate presigned URL for audio: {e} | job_id={job_id}"
)
output_payload["content"]["value"] = ""
output_payload["content"]["format"] = "url"
raw_content = copy.deepcopy(llm_call.content)
output = (
_resolve_llm_output(raw_content, project_id, session, job_id)
if isinstance(raw_content, dict)
else None
)

llm_response = LLMResponse(
provider_response_id=llm_call.provider_response_id or "",
conversation_id=llm_call.conversation_id,
provider=llm_call.provider,
model=llm_call.model,
output=output_payload,
output=output,
)

usage_payload = llm_call.usage
Expand Down
35 changes: 8 additions & 27 deletions backend/app/api/routes/llm_sts.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
from typing import Any, Literal
from uuid import UUID

from fastapi import APIRouter, Depends, HTTPException
from fastapi import APIRouter, Depends

from app.api.deps import AuthContextDep, SessionDep
from app.api.permissions import Permission, require_permission
Expand All @@ -18,7 +18,6 @@
from app.models.llm.request import (
ChainBlock,
ConfigBlob,
KaapiCompletionConfig,
LLMCallConfig,
LLMChainRequest,
QueryParams,
Expand All @@ -29,11 +28,9 @@
TextLLMParams,
TTSBlockSpec,
TTSLLMParams,
build_kaapi_completion_config,
)
from app.services.llm.chain.utils import (
DEFAULT_RAG_INSTRUCTIONS,
SUPPORTED_LANGUAGE_CODES,
)
from app.services.llm.chain.utils import DEFAULT_RAG_INSTRUCTIONS
from app.services.llm.jobs import start_chain_job
from app.utils import APIResponse, load_description, validate_callback_url

Expand Down Expand Up @@ -116,10 +113,10 @@ def _inline_call_config(
) -> LLMCallConfig:
return LLMCallConfig(
blob=ConfigBlob(
completion=KaapiCompletionConfig(
completion=build_kaapi_completion_config(
provider=provider,
type=type_,
params=params.model_dump(exclude_none=True),
params=params,
)
)
)
Expand Down Expand Up @@ -221,25 +218,9 @@ def speech_to_speech(
if request.callback_url:
validate_callback_url(str(request.callback_url))

if (
request.input_language
and request.input_language not in SUPPORTED_LANGUAGE_CODES
):
raise HTTPException(
status_code=422,
detail=f"Unsupported input language code: {request.input_language}. Supported: {', '.join(SUPPORTED_LANGUAGE_CODES)}",
)

if request.output_language and (
request.output_language not in SUPPORTED_LANGUAGE_CODES
or request.output_language in ("auto", "unknown")
):
tts_supported = SUPPORTED_LANGUAGE_CODES - {"auto", "unknown"}
raise HTTPException(
status_code=422,
detail=f"Unsupported output language code: {request.output_language}. Supported: {', '.join(tts_supported)}",
)

# Code membership + the auto/unknown exclusion on output_language are now
# enforced by SpeechToSpeechRequest itself (STSLanguageCode Literal +
# validate_output_language), so FastAPI 422s before this handler runs.
input_lang, output_lang = _resolve_languages(request)

blocks = [
Expand Down
6 changes: 5 additions & 1 deletion backend/app/core/langfuse/langfuse.py
Original file line number Diff line number Diff line change
Expand Up @@ -358,7 +358,11 @@ def langfuse_call(fn, *args, **kwargs):
as_type="generation",
name=f"{completion_config.provider}-completion",
input=query.input,
model=completion_config.params.get("model"),
model=(
completion_config.params.get("model")
if isinstance(completion_config.params, dict)
else getattr(completion_config.params, "model", None)
),
)

response: LLMCallResponse | None
Expand Down
8 changes: 7 additions & 1 deletion backend/app/crud/assessment/batch.py
Original file line number Diff line number Diff line change
Expand Up @@ -414,7 +414,13 @@ def submit_assessment_batch(
completion = config_blob.completion
provider_name = completion.provider or "openai"

params = dict(completion.params)
# Native params are a plain dict; Kaapi params are now a typed submodel.
raw_params = completion.params
params = (
dict(raw_params)
if isinstance(raw_params, dict)
else raw_params.model_dump(exclude_none=True)
)
Comment on lines +417 to +423

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== Locate file =="
git ls-files | rg '(^|/)backend/app/crud/assessment/batch\.py$|TextLLMParams|completion\.params|model_dump' || true

echo
echo "== Relevant snippets =="
if [ -f backend/app/crud/assessment/batch.py ]; then
  nl -ba backend/app/crud/assessment/batch.py | sed -n '380,450p'
fi

echo
echo "== Search TextLLMParams definition/usages =="
rg -n "class TextLLMParams|TextLLMParams|model_dump\\(|params\\.model_dump" backend/app -S || true

echo
echo "== Pydantic/model validation =="
rg -n "from pydantic import|import pydantic|BaseModel|Field\\(" backend/app -S || true

Repository: ProjectTech4DevAI/kaapi-backend

Length of output: 290


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== File exists and line range =="
head -n 450 backend/app/crud/assessment/batch.py | tail -n 80

echo
echo "== Locate TextLLMParams definition/usages =="
grep -RIn "class TextLLMParams\|TextLLMParams\|model_dump\|completion\.params" backend/app -S || true

echo
echo "== Pydantic deps/framework snippets =="
grep -RIn "from pydantic import\|import pydantic\|BaseModel\|Field(" backend/app pyproject.toml requirements*.txt poetry.lock -S 2>/dev/null || true

Repository: ProjectTech4DevAI/kaapi-backend

Length of output: 3254


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== TextLLMParams definition/usages =="
grep -RIn "class TextLLMParams\|TextLLMParams\|model_dump\|completion\.params" backend/app || true

echo
echo "== Pydantic deps/framework snippets =="
grep -RIn "from pydantic import\|import pydantic\|BaseModel\|Field(" backend/app pyproject.toml requirements*.txt poetry.lock 2>/dev/null || true

echo
echo "== Behavioral probe for Pydantic exclude_none vs exclude_unset =="
python3 - <<'PY'
import sys
print("python", sys.version)
try:
    import pydantic
    from pydantic import BaseModel, Field
    print("pydantic", pydantic.__version__)
except Exception as e:
    print("pydantic unavailable:", type(e).__name__, str(e))
    sys.exit(0)

class Params(BaseModel):
    temperature: float = 0.1
    top_p: float | None = None

p = Params()
print("exclude_none:", p.model_dump(exclude_none=True))
print("exclude_unset:", p.model_dump(exclude_unset=True))
print("both:", p.model_dump(exclude_none=True, exclude_unset=True))
PY

Repository: ProjectTech4DevAI/kaapi-backend

Length of output: 50387


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== TextLLMParams definition =="
sed -n '1,90p' backend/app/models/llm/request.py

echo
echo "== Assessment model params default =="
sed -n '300,340p' backend/app/models/assessment.py

echo
echo "== Mapper behavior for unset temperature =="
sed -n '1,90p' backend/app/services/llm/mappers.py
sed -n '1,180p' backend/app/tests/services/llm/test_mappers.py

echo
echo "== Pydantic behavioral probe =="
python3 - <<'PY'
import sys
print("python", sys.version)
try:
    import pydantic
    from pydantic import BaseModel, Field
    print("pydantic", pydantic.__version__)
except Exception as e:
    print("pydantic unavailable:", type(e).__name__, str(e))
    sys.exit(0)

class Params(BaseModel):
    temperature: float = 0.1
    top_p: float | None = None

p = Params()
print("exclude_none:", p.model_dump(exclude_none=True))
print("exclude_unset:", p.model_dump(exclude_unset=True))
print("both:", p.model_dump(exclude_none=True, exclude_unset=True))
PY

Repository: ProjectTech4DevAI/kaapi-backend

Length of output: 13820


🏁 Script executed:

#!/bin/bash
set -euo pipefail

python3 - <<'PY'
from pydantic import BaseModel, Field

class Params(BaseModel):
    temperature: float | None = 0.1
    top_p: float | None = None

p = Params()
print("exclude_none:", p.model_dump(exclude_none=True))
print("exclude_unset:", p.model_dump(exclude_unset=True))
print("both:", p.model_dump(exclude_none=True, exclude_unset=True))
PY

Repository: ProjectTech4DevAI/kaapi-backend

Length of output: 290


🌐 Web query:

Pydantic V2 model_dump exclude_none exclude_unset defaults unset fields

💡 Result:

In Pydantic V2, the model_dump method (and model_dump_json) provides specific parameters to control which fields are included in the serialized output. These parameters are essential for tasks like partial updates (PATCH requests) where you only want to send fields that the user explicitly provided [1][2]. Key Parameters for Field Exclusion: exclude_unset: When set to True, this excludes any fields that were not explicitly set during model instantiation [1][3]. Pydantic tracks which fields were provided using the model_fields_set property [1][4]. If you instantiate a model and only provide one field, only that field will appear in the output when exclude_unset=True [1]. Note that if you modify a field on the instance after it has been created, it is considered set and will be included [1][4]. exclude_none: When set to True, this excludes any fields whose value is None, regardless of whether they were explicitly set or assigned a default [1][3]. This is useful for cleaning up outputs by removing null values [5]. exclude_defaults: When set to True, this excludes any fields whose value is equal to their default value (using the == equality operator) [1][4]. Distinction: exclude_unset focuses on the origin of the value (was it provided at creation?) [1]. exclude_none focuses on the actual value itself (is it None?) [1]. exclude_defaults focuses on whether the current value matches the defined default value [1]. These parameters can be used independently or in combination to precisely tailor the serialization output [1][2]. If a field is explicitly set to None, exclude_unset=True will keep it in the output, while exclude_none=True will remove it [5][6]. If you need to detect if a field was missing versus set to None, you can inspect the model_fields_set attribute directly [6].

Citations:


Preserve unset Kaapi params during batch normalization.

TextLLMParams.temperature defaults to 0.1, so exclude_none=True serializes it as an explicit temperature when the typed config omits it. That normalized temperature=0.1 then overrides provider/model defaults, unlike native dict params. Add exclude_unset=True here, and keep Temperature suppression checks keyed to provided values rather than fields with defaults.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@backend/app/crud/assessment/batch.py` around lines 417 - 423, Update the
typed-parameter normalization in the batch completion flow to call model_dump
with exclude_unset=True alongside exclude_none=True, preserving only explicitly
provided Kaapi values such as temperature. Ensure the nearby Temperature
suppression checks in the same normalization logic are based on provided values,
not default-populated fields, while leaving native dict handling unchanged.

params.pop("instructions", None)
params.pop("system_instruction", None)
if isinstance(system_instruction, str) and system_instruction.strip():
Expand Down
9 changes: 7 additions & 2 deletions backend/app/crud/evaluations/core.py
Original file line number Diff line number Diff line change
Expand Up @@ -597,8 +597,13 @@ def resolve_model_from_config(
f"(config_id={eval_run.config_id}, version={eval_run.config_version}): {error}"
)

# params is a dict, not a Pydantic model, so use dict access
model = config.completion.params.get("model")
# Native params are a plain dict; Kaapi params are now a typed submodel.
completion_params = config.completion.params
model = (
completion_params.get("model")
if isinstance(completion_params, dict)
else getattr(completion_params, "model", None)
)
if not model:
raise ValueError(
f"Config for evaluation {eval_run.id} does not contain a 'model' parameter"
Expand Down
13 changes: 11 additions & 2 deletions backend/app/crud/model_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,12 @@ def validate_blob_model_or_raise(session: Session, blob: ConfigBlob) -> None:

provider = _normalize_provider(raw_provider)

model_name = (completion.params or {}).get("model") or None
params = completion.params
model_name = (
params.get("model")
if isinstance(params, dict)
else getattr(params, "model", None)
) or None
if not model_name:
raise HTTPException(
status_code=400,
Expand All @@ -170,7 +175,11 @@ def validate_blob_model_or_raise(session: Session, blob: ConfigBlob) -> None:
)

if completion_type == "tts" and model_row is not None:
voice = (completion.params or {}).get("voice")
voice = (
params.get("voice")
if isinstance(params, dict)
else getattr(params, "voice", None)
)
voice_spec = (
model_row.config.get("voice")
if isinstance(model_row.config, dict)
Expand Down
5 changes: 5 additions & 0 deletions backend/app/models/llm/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,11 @@
ConfigBlob,
KaapiLLMParams,
KaapiCompletionConfig,
KaapiTextCompletionConfig,
KaapiSTTCompletionConfig,
KaapiTTSCompletionConfig,
ProxyCompletionConfig,
build_kaapi_completion_config,
NativeCompletionConfig,
LlmCall,
AudioContent,
Expand Down
32 changes: 32 additions & 0 deletions backend/app/models/llm/constants.py
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,38 @@ class Modality(StrEnum):
FILES = "FILES"


# BCP-47 language codes accepted by the speech-to-speech endpoint (STT input /
# TTS output). Single source of truth: `SUPPORTED_LANGUAGE_CODES` in
# `app/services/llm/chain/utils.py` derives from this via `get_args`.
Comment on lines +66 to +68

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Make the alias the documented source of truth.

This comment calls SUPPORTED_LANGUAGE_CODES authoritative while also stating that it is derived from STSLanguageCode. Since utils.py derives the set via get_args, describe this Literal as the source of truth to avoid future edits to the wrong declaration.

Suggested wording
-# TTS output). Single source of truth: `SUPPORTED_LANGUAGE_CODES` in
-# `app/services/llm/chain/utils.py` derives from this via `get_args`.
+# TTS output). This Literal is the single source of truth; `SUPPORTED_LANGUAGE_CODES`
+# in `app/services/llm/chain/utils.py` is derived from it via `get_args`.
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
# BCP-47 language codes accepted by the speech-to-speech endpoint (STT input /
# TTS output). Single source of truth: `SUPPORTED_LANGUAGE_CODES` in
# `app/services/llm/chain/utils.py` derives from this via `get_args`.
# BCP-47 language codes accepted by the speech-to-speech endpoint (STT input /
# TTS output). This Literal is the single source of truth; `SUPPORTED_LANGUAGE_CODES`
# in `app/services/llm/chain/utils.py` is derived from it via `get_args`.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@backend/app/models/llm/constants.py` around lines 66 - 68, Update the
comments above STSLanguageCode to identify this Literal alias as the single
source of truth for accepted speech-to-speech language codes, and state that
SUPPORTED_LANGUAGE_CODES is derived from it via get_args. Do not describe
SUPPORTED_LANGUAGE_CODES as authoritative.

STSLanguageCode = Literal[
"auto",
"unknown",
"en-IN",
"hi-IN",
"bn-IN",
"kn-IN",
"ml-IN",
"mr-IN",
"od-IN",
"pa-IN",
"ta-IN",
"te-IN",
"gu-IN",
"as-IN",
"ur-IN",
"ne-IN",
"kok-IN",
"ks-IN",
"sd-IN",
"sa-IN",
"sat-IN",
"mni-IN",
"brx-IN",
"mai-IN",
"doi-IN",
]


DEFAULT_STT_MODEL = "gemini-2.5-pro"
DEFAULT_TTS_MODEL = "gemini-2.5-flash-preview-tts"
DEFAULT_TTS_VOICE = "Kore"
Expand Down
Loading
Loading