Skip to content

feature: deploy em staging a cada merge na master, com smoke, e2e e helm rollback automático (s4.2) - #20

Merged
devin-ai-integration[bot] merged 6 commits into
masterfrom
devin/unt162-14-deploy-staging
Nov 5, 2026
Merged

devin-ai-integration[bot] merged 6 commits into
masterfrom
devin/unt162-14-deploy-staging

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Contexto

Plano s4.2 (Replace cf push with a GKE delivery pipeline): cada merge em master passa a implantar em staging. Depois que o job publish de ci.yml (s4.1) envia as imagens ao Artifact Registry, o novo job deploy-staging chama o workflow reutilizável .github/workflows/deploy-staging.yml, que faz:

helm upgrade --install axon-trader deploy/helm/axon-trader -f values-staging.yaml \
  --set global.image.registry=$GCP_ARTIFACT_REGISTRY --set <svc>.image.repository=<svc> \
  --set <svc>.image.digest=sha256:...   # digests vindos de publish (needs.publish.outputs.digests)
kubectl rollout status deploy/{discovery-server,trading-engine,trader-app,trader-ui}
ci/deploy/smoke.sh   # data-initializer, newman (ESArchSmokeTest), ci/tasks/e2e-test-both-sides.sh, GET $STAGING_URL/api/command/api
qualquer falha -> helm rollback <última revisão deployed> --wait ; exit 1
  • Gate: sem GCP_WORKLOAD_IDENTITY_PROVIDER, GCP_CI_SERVICE_ACCOUNT, GCP_ARTIFACT_REGISTRY (as mesmas de publish) e GKE_CLUSTER (projects/<p>/locations/<gke_cluster_location>/clusters/<gke_cluster_name>) o job emite ::notice:: + resumo e sai verde. Com o gate ligado, exige também STAGING_CLOUDSQL_INSTANCE_CONNECTION_NAME, STAGING_CLOUDSQL_PSA_CIDR e STAGING_URL (falha com ::error:: se faltar). Hoje nenhuma está definida, então o merge não implanta nada.
  • Auth: só WIF (google-github-actions/auth + get-gke-credentials, SA ci-deployer com roles/container.developer). Nenhuma chave JSON, nenhum segredo no repositório.
  • Actuator sem rota pública (CHG-12): engineURL/appURL do e2e são kubectl port-forward para os Services; uiURL é STAGING_URL. A rota pública é checada à parte (/api/command/api).
  • Slack: porta das notificações do Concourse (_passed_ :white_check_mark: / <!here> *FAILED* :x:) via secret SLACK_WEBHOOK_URL; sem o secret, apenas um notice.
  • Chart: novo image.digest por serviço; quando definido, axon-common.image renderiza <registry>/<repo>@sha256:... no lugar da tag. publish envia $REGISTRY/<svc>, enquanto o chart usa por padrão esarch/<svc>, então o deploy define <svc>.image.repository=<svc>.
  • trading-engine continua com 1 réplica + Recreate (decisão d-engine-replicas); a prova verifica isso.
  • ci/deploy/helm-deploy.sh é o mesmo script em GKE e em kind; ci/kind/verify-deploy-rollback.sh é a prova offline (reaproveita ci/kind/setup.sh, de onde kind_backing foi extraída).

Ainda pendente: a execução real em staging (não existe projeto GCP; nada foi aplicado em GCP nem em TAS). A prova de rollback é local em kind, não uma execução do GitHub Actions. Em kind as imagens são referenciadas por tag (imagens locais não têm digest de registry); a renderização por digest foi verificada com helm template (abaixo).

Como testar

# imagens locais (como em ci/kind/verify-axon-routing.sh)
./mvnw -B -DskipTests package jib:dockerBuild && docker build -t esarch/trader-ui:latest trader-app-ui
bash ci/kind/verify-deploy-rollback.sh
bash deploy/helm/scripts/validate.sh
shellcheck -x ci/deploy/*.sh ci/kind/verify-deploy-rollback.sh && actionlint

Saída: bash ci/kind/verify-deploy-rollback.sh (kind v0.29.0, node v1.33.1, helm v3.16.4; 5m45s no commit inicial, repetida no head final com as correções da revisão: 5m57s, EXIT 0; trechos)

1. Imagens boas: upgrade + rollout + smoke + e2e verdes (revisão 1)


== 1. good images: expect deploy + smoke + e2e to pass
== deploy: release axon-trader in axon-trader (currently deployed revision: none)
Release "axon-trader" does not exist. Installing it now.
NAME: axon-trader
REVISION: 1

  kubectl -n axon-trader rollout status deploy/discovery-server deploy/trading-engine deploy/trader-app deploy/trader-ui
== deploy: rollout status (timeout 600s each)
Waiting for deployment "discovery-server" rollout to finish: 0 of 1 updated replicas are available...
deployment "discovery-server" successfully rolled out
deployment "trading-engine" successfully rolled out
Waiting for deployment "trader-app" rollout to finish: 0 of 2 updated replicas are available...
Waiting for deployment "trader-app" rollout to finish: 1 of 2 updated replicas are available...
deployment "trader-app" successfully rolled out
deployment "trader-ui" successfully rolled out
== deploy: revision 1 rolled out:
discovery-server=esarch/discovery-server:latest
trading-engine=esarch/trading-engine:latest
trader-app=esarch/trader-app:latest
trader-ui=esarch/trader-ui:latest
...
[http://localhost:28081/info] shows 'HTTP/1.1 200 OK' (as expected).
200
[http://localhost:28080/info] shows 'HTTP/1.1 200 OK' (Expected).
200
[http://localhost:28082] (Trader-UI) shows 'HTTP/1.1 200 OK' (Expected).
The DATA initializer is reporting data initialisation = true.
200
[http://localhost:28080/command/api] shows 'HTTP/1.1 200 OK' (as expected).
INTEGRATION TESTS FINISHED - NO ERRORS ;D 
| Command routing (data-initializer) | pass |
| Postman smoke (newman): 10/10 requests, 23/23 assertions | pass |
| e2e-test-both-sides.sh | pass |
SMOKE PASS
DEPLOY OK: axon-trader revision 1
release: 1 deployed Install complete

2. Imagem quebrada de trader-app (JVM com opção inválida, CrashLoopBackOff): rollout falha e rollback automático para a revisão 1


== 2. broken trader-app image (crash loop): expect rollout failure + automatic rollback
== deploy: release axon-trader in axon-trader (currently deployed revision: 1)
== deploy: helm upgrade --install axon-trader deploy/helm/axon-trader --namespace axon-trader --create-namespace -f ci/kind/values-kind.yaml --set trader-app.image.tag=broken
Release "axon-trader" has been upgraded. Happy Helming!
NAME: axon-trader
LAST DEPLOYED: Thu Nov  5 18:43:18 2026
NAMESPACE: axon-trader
STATUS: deployed
REVISION: 2
  kubectl -n axon-trader rollout status deploy/discovery-server deploy/trading-engine deploy/trader-app deploy/trader-ui
== deploy: rollout status (timeout 240s each)
deployment "discovery-server" successfully rolled out
deployment "trading-engine" successfully rolled out
Waiting for deployment "trader-app" rollout to finish: 1 out of 2 new replicas have been updated...
error: timed out waiting for the condition
NAME                                READY   STATUS             RESTARTS      AGE     IP            NODE                          NOMINATED NODE   READINESS GATES
discovery-server-5cf6df876c-mv5tk   1/1     Running            0             4m26s   10.244.0.11   esarch-deploy-control-plane   <none>           <none>
mysql-77677df4d5-7sthr              1/1     Running            0             4m39s   10.244.0.5    esarch-deploy-control-plane   <none>           <none>
rabbitmq-5d65b6b65b-89wwl           1/1     Running            0             4m39s   10.244.0.6    esarch-deploy-control-plane   <none>           <none>
trader-app-5cb7b9cf4d-htf24         0/1     CrashLoopBackOff   5 (45s ago)   4m      10.244.0.13   esarch-deploy-control-plane   <none>           <none>
trader-app-9fc488bbd-gtlz7          1/1     Running            0             4m26s   10.244.0.10   esarch-deploy-control-plane   <none>           <none>
trader-app-9fc488bbd-pvs4b          1/1     Running            0             4m26s   10.244.0.12   esarch-deploy-control-plane   <none>           <none>
trader-ui-68d574576d-lq8ct          1/1     Running            0             4m26s   10.244.0.9    esarch-deploy-control-plane   <none>           <none>
trader-ui-68d574576d-r4pk2          1/1     Running            0             4m26s   10.244.0.8    esarch-deploy-control-plane   <none>           <none>
trading-engine-5c5586779c-wfnbq     1/1     Running            0             4m26s   10.244.0.7    esarch-deploy-control-plane   <none>           <none>
4m25s       Normal    Created             pod/trading-engine-5c5586779c-wfnbq      Created container: trading-engine
::error::rollout of trader-app failed for axon-trader; rolling back to revision 1
Rollback was a success! Happy Helming!
deployment "discovery-server" successfully rolled out
deployment "trading-engine" successfully rolled out
deployment "trader-app" successfully rolled out
deployment "trader-ui" successfully rolled out
== rollback: helm history
REVISION	UPDATED                 	STATUS    	CHART            	APP VERSION	DESCRIPTION     
1       	Thu Nov  5 18:42:53 2026	superseded	axon-trader-0.1.0	latest     	Install complete
2       	Thu Nov  5 18:43:18 2026	superseded	axon-trader-0.1.0	latest     	Upgrade complete
3       	Thu Nov  5 18:47:20 2026	deployed  	axon-trader-0.1.0	latest     	Rollback to 1   
discovery-server=esarch/discovery-server:latest
trading-engine=esarch/trading-engine:latest
trader-app=esarch/trader-app:latest
trader-ui=esarch/trader-ui:latest
DEPLOY FAILED (rollout of trader-app): rolled back axon-trader to revision 1 (ok)
helm-deploy.sh exit status: 1 (expected non-zero)
release: 3 deployed Rollback to 1

3. Imagem quebrada de trader-ui (sem index.html): rollout passa, e2e falha (UI 404) e rollback automático para a revisão 3

deployment "trader-app" successfully rolled out

== 3. broken trader-ui image (no index.html): rollout passes, e2e fails, expect automatic rollback
== deploy: release axon-trader in axon-trader (currently deployed revision: 3)
REVISION: 4
deployment "trader-ui" successfully rolled out
== deploy: revision 4 rolled out:
discovery-server=esarch/discovery-server:latest
trading-engine=esarch/trading-engine:latest
trader-app=esarch/trader-app:latest
trader-ui=esarch/trader-ui:broken
== smoke 3. ci/tasks/e2e-test-both-sides.sh (engineURL=http://localhost:28081 appURL=http://localhost:28080 uiURL=http://localhost:28082)
The Trading-engine URL for this e2e test is:  http://localhost:28081 
The Trader-app URL for this e2e test is:  http://localhost:28080 
The Trader-UI URL for this e2e test is:  http://localhost:28082 
The health check status is reporting that http://localhost:28081 is UP
The health check status is reporting that http://localhost:28080 is UP
200
[http://localhost:28081/info] shows 'HTTP/1.1 200 OK' (as expected).
200
[http://localhost:28080/info] shows 'HTTP/1.1 200 OK' (Expected).
Error. Trader-UI is unresponsive. Failed to show '200 OK' on [http://localhost:28082]
| Command routing (data-initializer) | pass |
| Postman smoke (newman): 10/10 requests, 23/23 assertions | pass |
| e2e-test-both-sides.sh | **FAIL** (exit 1) |
SMOKE FAIL
::error::smoke/e2e checks failed for axon-trader; rolling back to revision 3
deployment "discovery-server" successfully rolled out
deployment "trading-engine" successfully rolled out
deployment "trader-app" successfully rolled out
deployment "trader-ui" successfully rolled out
== rollback: helm history
REVISION	UPDATED                 	STATUS    	CHART            	APP VERSION	DESCRIPTION     
1       	Thu Nov  5 18:42:53 2026	superseded	axon-trader-0.1.0	latest     	Install complete
2       	Thu Nov  5 18:43:18 2026	superseded	axon-trader-0.1.0	latest     	Upgrade complete
3       	Thu Nov  5 18:47:20 2026	superseded	axon-trader-0.1.0	latest     	Rollback to 1   
4       	Thu Nov  5 18:47:20 2026	superseded	axon-trader-0.1.0	latest     	Upgrade complete
5       	Thu Nov  5 18:47:28 2026	deployed  	axon-trader-0.1.0	latest     	Rollback to 3   
discovery-server=esarch/discovery-server:latest
trading-engine=esarch/trading-engine:latest
trader-app=esarch/trader-app:latest
trader-ui=esarch/trader-ui:latest
DEPLOY FAILED (smoke/e2e checks): rolled back axon-trader to revision 3 (ok)
helm-deploy.sh exit status: 1 (expected non-zero)
release: 5 deployed Rollback to 3

4. Depois dos rollbacks: imagens boas, trading-engine com 1 réplica, smoke + e2e verdes


== 4. after the rollbacks: release history, images and smoke + e2e
REVISION	UPDATED                 	STATUS    	CHART            	APP VERSION	DESCRIPTION     
1       	Thu Nov  5 18:42:53 2026	superseded	axon-trader-0.1.0	latest     	Install complete
2       	Thu Nov  5 18:43:18 2026	superseded	axon-trader-0.1.0	latest     	Upgrade complete
3       	Thu Nov  5 18:47:20 2026	superseded	axon-trader-0.1.0	latest     	Rollback to 1   
4       	Thu Nov  5 18:47:20 2026	superseded	axon-trader-0.1.0	latest     	Upgrade complete
5       	Thu Nov  5 18:47:28 2026	deployed  	axon-trader-0.1.0	latest     	Rollback to 3   
NAME               READY   REPLICAS   IMAGE
discovery-server   1       1          esarch/discovery-server:latest
mysql              1       1          mysql:5.7
rabbitmq           1       1          rabbitmq:3.7-management
trader-app         2       2          esarch/trader-app:latest
trader-ui          2       2          esarch/trader-ui:latest
trading-engine     1       1          esarch/trading-engine:latest
[http://localhost:28080/command/api] shows 'HTTP/1.1 200 OK' (as expected).
INTEGRATION TESTS FINISHED - NO ERRORS ;D 
| Command routing (data-initializer) | pass |
| Postman smoke (newman): 10/10 requests, 23/23 assertions | pass |
| e2e-test-both-sides.sh | pass |
SMOKE PASS
PASS: good deploy green; crash-looping and 404ing images both rolled back automatically; smoke + e2e green after

real	5m45.215s
user	0m18.403s
sys	0m7.446s
EXIT 0

Digests no values de staging (helm template ... -f values-staging.yaml + overrides do workflow)

      1           image: "us-central1-docker.pkg.dev/axon-trader-staging-placeholder/axon-trader/discovery-server@sha256:cccc…"
      1           image: "us-central1-docker.pkg.dev/axon-trader-staging-placeholder/axon-trader/trader-app@sha256:bbbb…"
      1           image: "us-central1-docker.pkg.dev/axon-trader-staging-placeholder/axon-trader/trader-ui@sha256:dddd…"
      1           image: "us-central1-docker.pkg.dev/axon-trader-staging-placeholder/axon-trader/trading-engine@sha256:aaaa…"
      2     type: Recreate
      2   replicas: 1

deploy/helm/scripts/validate.sh, shellcheck, actionlint

==> default|staging|prod|kind|kind-gateway: helm lint --strict -> 1 chart(s) linted, 0 chart(s) failed
default/staging/prod: Summary: 28 resources found in 1 file - Valid: 28, Invalid: 0, Errors: 0, Skipped: 0
kind: Valid: 23 ... kind-gateway: Valid: 25, Invalid: 0, Errors: 0
shellcheck -x ci/deploy/*.sh ci/kind/verify-deploy-rollback.sh -> sem achados
actionlint .github/workflows/*.yml -> sem achados

Correções da revisão (Devin Review)

  • helm-deploy.sh lê o histórico do release antes do upgrade; só release: not found conta como primeira instalação. Qualquer outro erro aborta antes do upgrade (result=not-deployed), então um check que falhar sempre tem revisão para o rollback. Uma falha no helm history após o rollout não interrompe mais o fluxo antes do smoke/rollback.
  • O script grava result= em $GITHUB_OUTPUT (deployed, rolled-back-to-N, rollback-failed, no-rollback, not-deployed) e o alerta de falha no Slack diz o estado real (revertido para N, rollback falhou, nada a reverter, ou falhou antes do upgrade).
  • Depois de um rollback, o result= é gravado antes de qualquer helm history, e essas leituras não abortam mais o script; o Slack só diz "staging unchanged" quando o passo de deploy nem começou (skipped) ou parou antes do upgrade (not-deployed); um result vazio com o deploy iniciado vira "staging state unknown".
  • smoke.sh espera as projeções que a coleção Postman lê (order book da primeira empresa, portfolio do primeiro usuário) depois do data-initializer; uma repetição da prova falhou com 404 em Get OrderBooks for Company por essa corrida.
$ GITHUB_OUTPUT=/tmp/gho KUBE_CONTEXT=kind-does-not-exist bash ci/deploy/helm-deploy.sh; echo "exit $?"; cat /tmp/gho
::error::cannot read the helm history of axon-trader; not deploying
Error: Kubernetes cluster unreachable: context "kind-does-not-exist" does not exist
exit 1
result=not-deployed

$ GITHUB_OUTPUT=/tmp/gho5 bash ci/kind/verify-deploy-rollback.sh; cat /tmp/gho5   # head final
DEPLOY OK: axon-trader revision 1
DEPLOY FAILED (rollout of trader-app): rolled back axon-trader to revision 1 (ok)
DEPLOY FAILED (smoke/e2e checks): rolled back axon-trader to revision 3 (ok)
| Postman smoke (newman): 10/10 requests, 23/23 assertions | pass |
| e2e-test-both-sides.sh | pass |
PASS: good deploy green; crash-looping and 404ing images both rolled back automatically; smoke + e2e green after
result=deployed
result=rolled-back-to-1
result=rolled-back-to-3

Devin-Org: engineering

Link to Devin session: https://devin-gtm.devinenterprise.com/sessions/9c3900f4aa0640c8b5ee49ba07a3705c
Open in Devin Desktop: https://devin-gtm.devinenterprise.com/desktop/session/9c3900f4aa0640c8b5ee49ba07a3705c?variant=devin
Requested by: @matthewguerra-cog


Devin Review

Status Commit
⚪ Not started —

Run Devin Review

Enable auto-review to review your pull requests automatically.

Devin Review (Staging)


Devin Review

…automatic helm rollback (s4.2)

Co-Authored-By: Matthew Guerra <matthew.guerra@cognition.ai>
@devin-ai-integration

Copy link
Copy Markdown
Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration Bot and others added 3 commits November 5, 2026 18:50
…ed check can always roll back; report the deploy result

Co-Authored-By: Matthew Guerra <matthew.guerra@cognition.ai>
…or never deployed

Co-Authored-By: Matthew Guerra <matthew.guerra@cognition.ai>
…ds, so it does not race them after data-initializer

Co-Authored-By: Matthew Guerra <matthew.guerra@cognition.ai>
@devin-ai-integration

Copy link
Copy Markdown
Author

Answers to the two Devin Review flags not posted inline (head 7bd0f04):

  • Some master merges bypass staging deployment (ci.yml paths-ignore): doesn't apply. Pushes that only touch infra/**, docs/** or *.md build no new images, so there is nothing new to deploy; staging deploys the images publish just pushed. Chart changes under deploy/helm/** (other than README) still trigger CI and the deploy. Infra changes go through the Terraform workflow, not Helm.
  • Failure alert overstates rollback status: fixed in 8da1c17. ci/deploy/helm-deploy.sh writes result= to $GITHUB_OUTPUT (deployed, rolled-back-to-N, rollback-failed, no-rollback, not-deployed) and the Slack failure text now says which: rolled back to revision N, rollback FAILED, no earlier revision, or failed before the helm upgrade (auth/credentials failures leave staging unchanged). Kind proof rerun on the head emitted result=deployed, result=rolled-back-to-1, result=rolled-back-to-3.

Also fixed in 7bd0f04 (found while rerunning the proof): ci/deploy/smoke.sh now waits for the query projections the Postman collection reads (first company's order book, first user's portfolio) after data-initializer; one rerun hit a 404 on Get OrderBooks for Company from that race.

Written by Devin

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration Bot and others added 2 commits November 5, 2026 19:13
… unreachable cluster still reports rollback-failed

Co-Authored-By: Matthew Guerra <matthew.guerra@cognition.ai>
…ed; otherwise state unknown

Co-Authored-By: Matthew Guerra <matthew.guerra@cognition.ai>
@devin-ai-integration
devin-ai-integration Bot merged commit 8d22a2d into master Nov 5, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant