Skip to content

UNT162-10: Eureka in-cluster (discovery-server) no lugar do p-service-registry para o roteamento do Axon - #17

Merged
devin-ai-integration[bot] merged 3 commits into
masterfrom
devin/UNT162-10-eureka-axon-routing
Nov 5, 2026
Merged

devin-ai-integration[bot] merged 3 commits into
masterfrom
devin/UNT162-10-eureka-axon-routing

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Summary

Plano s3.3 (UNT162-10): troca o p-service-registry do TAS pelo Eureka in-cluster do próprio discovery-server, como decidido no ticket. O roteamento do Axon não muda: o SpringCloudCommandRouter continua publicando as capabilities de cada pod nos metadados da instância Eureka (loadFactor, serializedCommandFilter), com axon.distributed.spring-cloud.fallback-to-http-get=false. Boot 2.0.3 / Java 8 / Axon 3.3.3 mantidos.

Contexto

  • pom.xml: spring-cloud-services-starter-service-registry → spring-cloud-starter-netflix-eureka-client (Finchley); BOM spring-cloud-services-dependencies removido.
  • discovery-server/.../application-kubernetes.properties: Eureka standalone, sem self-preservation, eviction a cada 5 s e sem read-only cache, para o anel de consistent hash não manter um pod morto.
  • trader-app / trading-engine (application-kubernetes.properties): lease 5 s / expiração 15 s, fetch e replicação a cada 5 s, prefer-ip-address=true (IP do pod). Pod que cai sem deregistrar sai do anel em ~20-30 s (medido: 28 s e 33 s).
  • deploy/kubernetes/discovery-server/: Deployment (non-root, rootfs read-only), Service ClusterIP discovery-server:8761 e NetworkPolicies (apps → Eureka:8761; app ↔ app:8080 para o SpringHttpCommandBusConnector, já que infra/k8s/platform nega ingress por padrão). Incluído no deploy/kubernetes/kustomization.yaml para o chart do s3.5.
  • CommandLoggingConfiguration (trading-engine): interceptor no localSegment que loga Handling command <payload>; é a evidência de qual pod tratou cada comando.
  • ci/kind/: harness reproduzível (verify-axon-routing.sh) — kind, MySQL/RabbitMQ descartáveis com senhas geradas em runtime (nada commitado), 2× trader-app e 2× trading-engine.
  • docs/migration/tas-gke-mapping.md: CHG-1, CHG-3, CHG-6, APP-4, SVC-4 e OPEN-5 fechados com a opção A.

Contrato de API: sem mudança (nenhum endpoint, payload ou evento alterado).

Nada foi aplicado em GCP nem em apps/serviços do TAS.

Como testar

JAVA_HOME=<jdk8> ./mvnw -B -DskipTests package jib:dockerBuild   # imagens esarch/*:latest
./ci/kind/verify-axon-routing.sh            # ESARCH_KIND_KEEP=1 para manter o cluster

O script falha (exit ≠ 0) se qualquer verificação abaixo não passar:

  1. Eureka lista 4 instâncias com metadados do Axon.
  2. 40 ordens buy/sell com 2 trading-engines: cada transactionId com HTTP 200, exatamente 1 linha Handling command em todos os pods e exatamente 1 *TransactionStartedEvent no event store; cada agregado em um único pod (consistent hash).
  3. kill -9 no JVM do trading-engine mais ocupado, pelo node (sem shutdown hook → sem deregistro), e ao mesmo tempo scale --replicas=1 com pod-deletion-cost para o ReplicaSet remover o pod morto — senão o kubelet reinicia o container no mesmo IP e o JVM novo pode se re-registrar com o mesmo instance id antes do lease expirar (aconteceu numa rodada). Um watcher em background mede a evicção desde o kill e falha se o lease for re-registrado ou sumir em < 10 s (= deregistro, não crash).
    Ordens enviadas enquanto o Eureka ainda lista o pod morto, por ordem: HTTP 200 ⇔ exatamente 1 linha de log e 1 TransactionStartedEvent; qualquer outro status ⇔ 0 linhas e 0 eventos (evento persistido implica tratado; falha sem efeito colateral). Se um timeout HTTP chegasse depois de o comando ser tratado, o script falharia — não é aceito.
  4. Após a evicção: 40 ordens só no sobrevivente, exatamente uma vez, incluindo agregados que eram do pod morto.
  5. scale --replicas=2: um pod novo entra no anel e as 40 ordens seguintes se dividem de novo entre os dois.
  6. Event store inteiro: pares (agregado, sequence) duplicados e transações com mais de um TransactionStarted são contados e qualquer valor ≠ 0 falha o script.
  7. Depois do primeiro apply (filtrado por app.kubernetes.io/name notin (trader-app,trading-engine), que também seleciona objetos sem o label), o script confere que mysql-initdb, trader-app-config e trading-engine-config existem antes do rollout do MySQL.

Evidência

Rodada em cluster que não existia (sem ESARCH_KIND_KEEP), no head deste PR, kind v0.24.0 / node v1.31.0:

kind delete cluster --name esarch-routing; kind get clusters   # vazio
./ci/kind/verify-axon-routing.sh

Saída (sem as linhas created/Waiting for rollout do kubectl e o banner do kind):

clusters before run: []
== 1. kind cluster 'esarch-routing', images, namespace and throwaway secrets
Creating cluster "esarch-routing" ...
loaded esarch/discovery-server:latest
loaded esarch/trader-app:latest
loaded esarch/trading-engine:latest
loaded mysql:5.7
pulled rabbitmq:3.7-management on the node
== 2. Deploy discovery-server, MySQL, RabbitMQ, then 2x trading-engine and 2x trader-app (profile kubernetes)
NAME                                READY   STATUS    RESTARTS      AGE    IP            NODE                           NOMINATED NODE   READINESS GATES
discovery-server-7d659967d9-x9lrp   1/1     Running   0             2m7s   10.244.0.5    esarch-routing-control-plane   <none>           <none>
mysql-6456b68dc7-zwjw4              1/1     Running   0             2m7s   10.244.0.6    esarch-routing-control-plane   <none>           <none>
rabbitmq-7dbdc9d6dd-2m84t           1/1     Running   0             2m7s   10.244.0.7    esarch-routing-control-plane   <none>           <none>
trader-app-7d74f8f6c7-7gnfh         1/1     Running   1 (87s ago)   101s   10.244.0.10   esarch-routing-control-plane   <none>           <none>
trader-app-7d74f8f6c7-qqw28         1/1     Running   2 (66s ago)   101s   10.244.0.8    esarch-routing-control-plane   <none>           <none>
trading-engine-74d4fb8c8f-2wr6t     1/1     Running   2 (65s ago)   101s   10.244.0.9    esarch-routing-control-plane   <none>           <none>
trading-engine-74d4fb8c8f-jvl4p     1/1     Running   2 (62s ago)   101s   10.244.0.11   esarch-routing-control-plane   <none>           <none>
Eureka registry:
TRADER-APP trader-app-7d74f8f6c7-7gnfh:trader-app ip=10.244.0.10:8080 status=UP axon.loadFactor=100 axon.commandFilter=yes
TRADER-APP trader-app-7d74f8f6c7-qqw28:trader-app ip=10.244.0.8:8080 status=UP axon.loadFactor=100 axon.commandFilter=yes
TRADING-ENGINE trading-engine-74d4fb8c8f-2wr6t:trading-engine ip=10.244.0.9:8080 status=UP axon.loadFactor=100 axon.commandFilter=yes
TRADING-ENGINE trading-engine-74d4fb8c8f-jvl4p:trading-engine ip=10.244.0.11:8080 status=UP axon.loadFactor=100 axon.commandFilter=yes
== 3. Seed demo data through trader-app (needs a trading-engine route: CommandRouter.findDestination)
data-initializer: true
order book 3cb32ff8-1432-448b-927f-e98675c0804b, 9 portfolios
== 4. Orders with two trading-engine replicas
[before] 40 orders: 40 handled exactly once, 0 not
[before] start-transaction commands per pod: trading-engine-74d4fb8c8f-2wr6t 17  trading-engine-74d4fb8c8f-jvl4p 23  
aggregates handled: 52, handled by more than one pod: 0
sample of command handling across pods:
  trading-engine-74d4fb8c8f-2wr6t: 2026-11-05 16:47:39.001  INFO [trading-engine,fa12c0aab623ed83,a97932de7af4d01c,false] 1 --- [nio-8080-exec-2] i.p.r.c.t.t.CommandLoggingConfiguration  : Handling command CreateOrderBoo
  trading-engine-74d4fb8c8f-2wr6t: 2026-11-05 16:47:39.275  INFO [trading-engine,fa12c0aab623ed83,0b8135e09d6f76c6,false] 1 --- [nio-8080-exec-4] i.p.r.c.t.t.CommandLoggingConfiguration  : Handling command CreateOrderBoo
  trading-engine-74d4fb8c8f-jvl4p: 2026-11-05 16:47:39.362  INFO [trading-engine,fa12c0aab623ed83,acb06834a2431650,false] 1 --- [nio-8080-exec-2] i.p.r.c.t.t.CommandLoggingConfiguration  : Handling command CreateOrderBoo
  trading-engine-74d4fb8c8f-jvl4p: 2026-11-05 16:47:39.710  INFO [trading-engine,fa12c0aab623ed83,2ba48fac9cacdb39,false] 1 --- [nio-8080-exec-3] i.p.r.c.t.t.CommandLoggingConfiguration  : Handling command CreatePortfoli
== 5. Kill the busiest trading-engine pod (SIGKILL, no Eureka deregistration)
victim=trading-engine-74d4fb8c8f-jvl4p (10.244.0.11) owned 28 aggregates; survivor=trading-engine-74d4fb8c8f-2wr6t
kill -9 8247 (java in trading-engine-74d4fb8c8f-jvl4p)
== 5a. Orders while Eureka still lists the dead pod: at most once, and HTTP 200 only if handled
[during-eviction] 8 orders: 8 handled exactly once (HTTP 200), 0 rejected and not handled, 0 duplicates
Eureka evicted trading-engine-74d4fb8c8f-jvl4p after 33 s (lease expiry, not a deregistration); registry now:
TRADER-APP trader-app-7d74f8f6c7-7gnfh:trader-app ip=10.244.0.10:8080 status=UP axon.loadFactor=100 axon.commandFilter=yes
TRADER-APP trader-app-7d74f8f6c7-qqw28:trader-app ip=10.244.0.8:8080 status=UP axon.loadFactor=100 axon.commandFilter=yes
TRADING-ENGINE trading-engine-74d4fb8c8f-2wr6t:trading-engine ip=10.244.0.9:8080 status=UP axon.loadFactor=100 axon.commandFilter=yes
== 6. Orders with one trading-engine replica left (deployment scaled to 1)
[after-kill] 40 orders: 40 handled exactly once, 0 not
[after-kill] start-transaction commands per pod: trading-engine-74d4fb8c8f-2wr6t 40  
aggregates owned by trading-engine-74d4fb8c8f-jvl4p before the kill and handled by trading-engine-74d4fb8c8f-2wr6t after it: 2
== 7. Scale back to 2: a replacement pod joins the ring
TRADER-APP trader-app-7d74f8f6c7-7gnfh:trader-app ip=10.244.0.10:8080 status=UP axon.loadFactor=100 axon.commandFilter=yes
TRADER-APP trader-app-7d74f8f6c7-qqw28:trader-app ip=10.244.0.8:8080 status=UP axon.loadFactor=100 axon.commandFilter=yes
TRADING-ENGINE trading-engine-74d4fb8c8f-2wr6t:trading-engine ip=10.244.0.9:8080 status=UP axon.loadFactor=100 axon.commandFilter=yes
TRADING-ENGINE trading-engine-74d4fb8c8f-z6gpl:trading-engine ip=10.244.0.12:8080 status=UP axon.loadFactor=100 axon.commandFilter=yes
[rejoined] 40 orders: 40 handled exactly once, 0 not
[rejoined] start-transaction commands per pod: trading-engine-74d4fb8c8f-2wr6t 29  trading-engine-74d4fb8c8f-z6gpl 11  
== 8. Whole event store: duplicates and totals
duplicate (aggregate, sequence) pairs: 0
TransactionStarted events per transaction > 1: 0
BuyOrderPlacedEvent                        45
BuyTransactionConfirmedEvent               45
BuyTransactionExecutedEvent                16
BuyTransactionStartedEvent                 64
CashDepositedEvent                         21
CashReservationCancelledEvent              7
CashReservationConfirmedEvent              16
CashReservedEvent                          64
ItemReservationConfirmedForPortfolioEvent  12
ItemsAddedToPortfolioEvent                 25
ItemsReservedEvent                         63
OrderBookCreatedEvent                      3
PortfolioCreatedEvent                      9
SellOrderPlacedEvent                       38
SellTransactionConfirmedEvent              38
SellTransactionExecutedEvent               12
SellTransactionStartedEvent                64
TradeExecutedEvent                         16
orders posted: 128, HTTP 200: 128
PASS: two replicas of each service, every buy/sell handled exactly once, routing survived killing trading-engine-74d4fb8c8f-jvl4p
real	8m6.799s
EXIT 0

Mudança só em ci/kind/verify-axon-routing.sh desde o head anterior (nenhum código Java); ./mvnw -B test (Java 8, Maven 3.5.4) no head anterior: 171 testes, 0 falhas, 0 erros.

Nesta rodada as 8 ordens da janela de evicção receberam HTTP 200 (roteadas ao sobrevivente). Numa rodada anterior 1 de 8 foi roteada ao pod morto: HTTP 500 após o timeout de conexão TCP (~2 min, o IP do pod já não existe), 0 linhas de log e 0 eventos — o caso “não tratado” acima. É o comportamento do DistributedCommandBus sem retry; um RetryScheduler no CommandGateway reduziria essa janela, fora do escopo deste passo.

Observação: no kind single-node alguns starts de trader-app/trading-engine falham com UnknownHostException: rabbitmq/mysql (CoreDNS sob carga de 4 JVMs) e o Kubernetes reinicia o container (coluna RESTARTS); o kubectl port-forward do script se re-estabelece quando isso derruba o pod de destino. Não afeta o roteamento.

Devin-Org: engineering

Link to Devin session: https://devin-gtm.devinenterprise.com/sessions/a98baf9b401546f9a23b947082d04865
Open in Devin Desktop: https://devin-gtm.devinenterprise.com/desktop/session/a98baf9b401546f9a23b947082d04865?variant=devin
Requested by: @matthewguerra-cog


Devin Review

…-registry para o roteamento do Axon, com prova em kind (2 réplicas, kill -9)

Co-Authored-By: Matthew Guerra <matthew.guerra@cognition.ai>
@devin-ai-integration

Copy link
Copy Markdown
Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Devin Review: 3 flags

Not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

Devin Review

…st-once window, duplicate checks fail)

- assert the generated ConfigMaps exist after the first (filtered) apply
- during the eviction window: HTTP 200 <=> one log line and one event; other status <=> none
- duplicate event counts now fail the script
- kill -9 plus scale to 1 (pod-deletion-cost) so the restarted container cannot re-register before the lease expires; eviction timed by a background watcher
- self-healing port-forwards

Co-Authored-By: Matthew Guerra <matthew.guerra@cognition.ai>
@devin-ai-integration

Copy link
Copy Markdown
Author

Resposta às 3 flags do Devin Review, todas no head a5ce30f (só ci/kind/verify-axon-routing.sh mudou):

  1. ANALYSIS_0001 — mysql-initdb no primeiro apply. O seletor app.kubernetes.io/name notin (trader-app,trading-engine) também seleciona objetos sem o label, então os ConfigMaps gerados entram no primeiro passo (configmap/mysql-initdb created aparece já ali). Para não depender disso implicitamente, o script agora confere mysql-initdb, trader-app-config e trading-engine-config logo depois do primeiro apply e falha se faltar algum. Prova: kind delete cluster --name esarch-routing → kind get clusters vazio → ./ci/kind/verify-axon-routing.sh sem ESARCH_KIND_KEEP → EXIT 0 (saída completa na descrição do PR, seção Evidência).
  2. ANALYSIS_0002 — check_at_most_once. Agora é bicondicional por ordem: HTTP 200 ⇔ log_lines=1 e started_events=1; qualquer outro status ⇔ log_lines=0 e started_events=0. Um não-200 com evento persistido, ou um 200 sem linha de log, falha o script. Também foi corrigida uma corrida que surgiu nas rodadas a frio: o kubelet reiniciava o container morto no mesmo IP e o JVM novo podia se re-registrar antes do lease expirar. Agora o kill é acompanhado de scale --replicas=1 com pod-deletion-cost, e um watcher em background falha se o lease for re-registrado ou sumir em < 10 s (= deregistro). Evicção medida: 33 s.
  3. ANALYSIS_0003 — duplicatas. As duas contagens ficam em variáveis, são impressas e [ "$dup_seq" = 0 ] && [ "$dup_started" = 0 ] || fail ... vem antes do PASS.

Written by Devin

Co-Authored-By: Matthew Guerra <matthew.guerra@cognition.ai>
@devin-ai-integration
devin-ai-integration Bot merged commit 0b2f9c6 into master Nov 5, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant