feat(release): Move Docker image to ECR and build it on EC2

[Ayush8923]

Issue:

Closes ProjectTech4DevAI/kaapi-backend#1056

Summary

• Update the deployment script because we move the docker image in ECR and then pull the image from ECR and build it on EC2 instance.

1. guardrails configure crashed on a Hub blip

guardrails configure writes ~/.guardrailsrc and then calls the Hub /auth endpoint only to validate the token. The Hub sits behind CloudFront and intermittently returns a non-JSON body; the CLI turns that into a hard sys.exit(1), which kills the step under set -euo pipefail.

(configure is actually written to tolerate auth failures, but fetch() exits the process before that handler can run.)

Fix: write ~/.guardrailsrc directly instead of shelling out to guardrails configure — same file, same contents, minus a network round-trip we don't need. Verified the file round-trips through Guardrails' own RC.load(). Also added retry-with-backoff around guardrails hub install so a genuine Hub hiccup doesn't fail the build.

2. nsfw_text post-install blocked by nltk 3.10.1

nltk 3.10.1 (new release, pulled in unpinned by the nsfw_text validator) added an import hook that blocks imports resolving under the current working directory, as anti-hijacking protection. Our venv is at backend/.venv —
inside the working directory — so every dependency nltk imports (regex joblib, …) trips the check and the post-install script dies.

Checklist

Before submitting a pull request, please ensure that you mark these task.

• Ran fastapi run --reload app/main.py or docker compose up in the repository root and test.
• If you've fixed a bug or added code that is tested and has test cases.

[coderabbitai]

📝 Walkthrough

Walkthrough

Production and staging workflows now build and push backend images to Amazon ECR. EC2 deployments pull those images through Docker Compose. Production triggers accept version-like tags, and SSM output retrieval is centralized across terminal states. CI and Guardrails installation now disable NLTK import security checks and retry validator installation.

Changes

ECR deployment flow

Layer / File(s)	Summary
ECR image reference docker-compose.yml	The prestart and backend services now use ECR registry, repository, and image-tag variables.
Production image build and deployment .github/workflows/cd-production.yml	Production accepts version-like tags, publishes the backend image to ECR, and deploys the selected tag to EC2. EC2 pulls the image, runs the prestart profile, starts the Compose stack, and retrieves SSM output through a shared helper.
Staging image build and deployment .github/workflows/cd-staging.yml	Staging publishes the backend image to ECR, pulls it on EC2, runs the prestart service, starts the Compose stack, and centralizes SSM output retrieval.

Guardrails installation setup

Layer / File(s)	Summary
NLTK and Guardrails configuration .github/workflows/continuous-integration.yml, backend/Dockerfile, backend/scripts/install_guardrails_from_hub.sh	CI, the backend image, and the installation script disable NLTK import security checks. The script writes Guardrails configuration directly.
Validator installation retries backend/scripts/install_guardrails_from_hub.sh	The script retries commands with exponential backoff and retries hub validator installation up to four times.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant GitHubActions
  participant ECR
  participant SSM
  participant EC2
  GitHubActions->>ECR: Build and push backend image
  GitHubActions->>SSM: Send deployment command
  SSM->>EC2: Authenticate and pull backend image
  EC2->>EC2: Run prestart profile and start Compose stack
  GitHubActions->>SSM: Poll status and retrieve output

Loading

Possibly related PRs

• ProjectTech4DevAI/kaapi-guardrails#95: Modifies the staging EC2 deployment workflow and SSM polling.
• ProjectTech4DevAI/kaapi-guardrails#100: Introduces the production deployment workflow updated here.
• ProjectTech4DevAI/kaapi-guardrails#138: Modifies Docker cleanup commands in the staging and production workflows.

Suggested reviewers: nishika26

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The workflows build and push the image to ECR, remove EC2-side builds, and pull and run the ECR image during deployment for #1056.
Out of Scope Changes check	✅ Passed	The changes support ECR deployment and Guardrails image installation; no unrelated code changes are evident.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the main release change: moving the Docker image to Amazon ECR and changing where the image is built.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/move-docker-image-to-ecr

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/cd-production.yml:
- Around line 22-25: Disable persisted checkout credentials in both deploy
workflows: add persist-credentials: false to the existing with block for
actions/checkout in .github/workflows/cd-production.yml at lines 22-25, and add
a with block containing persist-credentials: false to the checkout step in
.github/workflows/cd-staging.yml at lines 21-22.

In @.github/workflows/cd-staging.yml:
- Around line 34-49: Update the “Build and Push Docker Image” and “Trigger
deploy on EC2 via SSM” steps to use github.sha as the Docker image tag instead
of latest. Ensure the image is both pushed and referenced through IMAGE_TAG
using that same commit SHA.
- Line 51: Update the DEPLOY_CMD deployment sequence to reset the working tree
to origin/main after git fetch origin, instead of the potentially stale local
main branch; preserve the existing hard reset and subsequent deployment
commands.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 39ba7ab5-0fa8-4f79-8299-04dfb213d0d0

📥 Commits

Reviewing files that changed from the base of the PR and between 4857af1 and 3fbeeb1.

📒 Files selected for processing (3)

• .github/workflows/cd-production.yml
• .github/workflows/cd-staging.yml
• docker-compose.yml

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

.github/workflows/cd-production.yml (1)

43-44: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Align both standalone Docker builds with the Compose build secrets.

backend/Dockerfile reads HF_TOKEN and GUARDRAILS_HUB_API_KEY via BuildKit secret mounts, but both workflows build ./backend without passing those secrets and neither workflow exposes them. Pass equivalent BuildKit secrets in production and staging, or validate that their absence is intentional.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/cd-production.yml around lines 43 - 44, Update the
standalone Docker build commands in .github/workflows/cd-production.yml lines
43-44 and .github/workflows/cd-staging.yml lines 39-40 to pass BuildKit secrets
for HF_TOKEN and GUARDRAILS_HUB_API_KEY, matching the Compose build
configuration; if these secrets are intentionally unavailable, explicitly
validate and document that absence instead.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In @.github/workflows/cd-production.yml:
- Around line 43-44: Update the standalone Docker build commands in
.github/workflows/cd-production.yml lines 43-44 and
.github/workflows/cd-staging.yml lines 39-40 to pass BuildKit secrets for
HF_TOKEN and GUARDRAILS_HUB_API_KEY, matching the Compose build configuration;
if these secrets are intentionally unavailable, explicitly validate and document
that absence instead.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 12f583ed-45ff-4aae-adc3-e6cc9c9cc392

📥 Commits

Reviewing files that changed from the base of the PR and between 3fbeeb1 and 01b7c3d.

📒 Files selected for processing (2)

• .github/workflows/cd-production.yml
• .github/workflows/cd-staging.yml

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/continuous-integration.yml:
- Around line 12-13: Remove the global NLTK_DISABLE_IMPORT_SECURITY environment
setting from .github/workflows/continuous-integration.yml lines 12-13,
backend/Dockerfile lines 50-51, and
backend/scripts/install_guardrails_from_hub.sh lines 11-12; no direct changes
are needed elsewhere, and retain security controls unless a specific supported
NLTK call requires a scoped override.

In `@backend/scripts/install_guardrails_from_hub.sh`:
- Around line 42-57: Update the configuration flow around the Guardrails
installation script so GUARDRAILS_HUB_API_KEY is never persisted in the image
layer or `${HOME}/.guardrailsrc`; use a temporary HOME or tmpfs for build-time
setup, or defer configuration creation until container startup using a runtime
secret. Preserve the existing metrics, remote-inferencing, and anonymous-ID
settings while ensuring any build-time secret-backed files are removed before
the step completes.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: c9c13111-9963-43d6-9b07-c759b86c7760

📥 Commits

Reviewing files that changed from the base of the PR and between 01b7c3d and 3e8e157.

📒 Files selected for processing (5)

• .github/workflows/cd-production.yml
• .github/workflows/cd-staging.yml
• .github/workflows/continuous-integration.yml
• backend/Dockerfile
• backend/scripts/install_guardrails_from_hub.sh

🚧 Files skipped from review as they are similar to previous changes (2)

• .github/workflows/cd-staging.yml
• .github/workflows/cd-production.yml