Move to the internal load balancer behind an API Gateway

Summary

Move BuildBot from the testing Application Load Balancer to the internal one, and put an API Gateway, created by the stack, in front of it. Clients call the API Gateway, which forwards only the application's endpoints to the internal ALB listener, which routes to the ECS service as today.

Changes

1. Switch to the internal load balancer

In cloudformation.json, change Deployment.LoadBalancer.ListenerArn from

arn:aws:elasticloadbalancing:eu-west-1:117769150821:listener/app/testing/c383c5dd580a74f5/e82083ed48688c7a

to

arn:aws:elasticloadbalancing:eu-west-1:117769150821:listener/app/internal/fbd77c1a39a60f7f/e01c11e315c3ff6e

The ListenerRule, TargetGroup and the ECS service's LoadBalancers block stay in place, now attached to the internal listener.

2. Add an API Gateway that forwards to the internal load balancer

Add an API Gateway to cloudformation.json whose integration targets the internal ALB listener above, not the ECS tasks directly. Deploy one route for each endpoint in src/BuildBot/Helpers/Endpoints.*.cs:

Method	Path	Source
GET	`/ping`	`Endpoints.Test.cs`
POST	`/github/ping`	`Endpoints.GitHub.cs`
POST	`/github/push`	`Endpoints.GitHub.cs`
POST	`/github/status`	`Endpoints.GitHub.cs`
POST	`/watchtower/deploy`	`Endpoints.Watchtower.cs`
POST	`/cloudformation/deploy`	`Endpoints.CloudFormation.cs`

No catch-all ($default / {proxy+}) route. Keep any new per-service values in the Deployment mapping rather than parameter Defaults, per #417.

Notes / open questions

An internal ALB is reachable from API Gateway only through a VPC Link (HTTP API AWS::ApiGatewayV2::VpcLink, integration ConnectionType: VPC_LINK with the listener ARN as IntegrationUri). Does a shared VPC Link to the internal ALB already exist that should be reused, or should this stack create one (needs subnets and security groups)?
The listener rule matches on the host header (LoadBalancerHostName). Should the API Gateway integration override the Host header to that value (via request parameter mapping), or should the listener rule change to match on path instead?
Custom domain: should the stack create an API Gateway custom domain / API mapping for the public host name, and who updates DNS?

Networking guidance for the VPC Link (partly answers open question 1):

VPC link SG: egress 443 to the VPC is enough. alb-internal already allows 443 from 172.31.0.0/16.

So:

The VPC Link needs its own security group with a single egress rule: TCP 443 to 172.31.0.0/16. No ingress rules are needed.
alb-internal's security group needs no change, because it already allows 443 from 172.31.0.0/16.
The integration targets the internal listener on 443 (HTTPS).

Still open: whether this stack creates the VPC Link and its security group, or reuses an existing link.

Decision on open question 1: only BuildBot will sit behind API Gateway, so this stack creates its own VPC Link (AWS::ApiGatewayV2::VpcLink) and its security group (egress TCP 443 to 172.31.0.0/16, no ingress), using the service's existing subnets. No shared link is needed.

Decision on open question 2: the API Gateway integration sets the Host header to the LoadBalancerHostName value (HTTP API integration RequestParameters: overwrite:header.Host), so the existing host-header listener rule on the internal ALB stays unchanged.

Decision on open question 3: the stack also manages the public DNS. It creates:

an API Gateway custom domain (AWS::ApiGatewayV2::DomainName, regional endpoint) with an ACM certificate in eu-west-1;
an API mapping (AWS::ApiGatewayV2::ApiMapping) from that domain to the API's stage;
a Route 53 alias record (AWS::Route53::RecordSet, A and AAAA if wanted) pointing the host name at the custom domain's regional domain name / hosted zone ID.

Values needed in the Deployment mapping (still to be confirmed):

Public host name for the API (the same value as LoadBalancerHostName, or a different one?).
Route 53 hosted zone ID (or name) for that host name.
ACM certificate ARN in eu-west-1 covering that host name.

Caution: if a DNS record for that host name already exists outside this stack (for example, pointing at the testing ALB), CloudFormation will fail to create it. Delete it or import it into the stack as part of the cut-over.

Dependency, from the repository owner in live chat:

also #421 should depend on #420 too

Recorded: implementation of this issue waits until #420 (ECS secrets moved to Secrets, closing #419) merges, then branches from main. Both changes edit cloudformation.json, so this order avoids conflicting edits to the task definition.

Correction to the previous comment, which recorded the dependency as #420. The repository owner corrected this in live chat:

no not #420 #422

Corrected: implementation of this issue waits until the PR for #422 (GitHub webhook signature verification) merges, then branches from main. #422 itself waits for #420, so the order is #420, then #422, then #421.

Direction from the repository owner in live chat:

it needs to work with the other things recently gone in specifically #426 and the future ones for cloudformation and watchtower

This is reflected in the implementation plan below, together with the mappings rule recorded on #427 ("everything should be in mappings unless its explicitly set as a parameter in the deploy workflow").

Implementation Plan

Files to change

cloudformation.json:
- point ListenerRule at the internal listener;
- add the API Gateway HTTP API, VPC Link, VPC Link security group, integration, six routes, stage, custom domain, API mapping and Route 53 record;
- declare the ApiHost parameter, which the deploy action already passes;
- add the new values to the Deployment mapping.
CHANGELOG.md: entries via dotnet changelog, including a Deployment Changes entry for the cut-over steps.
README.md: in the Configuration section, note that webhooks now arrive through the API Gateway host.

Approach

Load balancer. Deployment.LoadBalancer.ListenerArn becomes the internal listener (.../app/internal/fbd77c1a39a60f7f/e01c11e315c3ff6e, the same one funfair-ethereum-proxy-server uses). ListenerRule, TargetGroup and the ECS service's LoadBalancers block stay as they are.

API Gateway. An HTTP API (AWS::ApiGatewayV2::Api) with:

a stack-owned AWS::ApiGatewayV2::VpcLink, using the existing SubnetIDs and a new security group with only egress TCP 443 to 172.31.0.0/16;
one HTTP_PROXY integration (ConnectionType: VPC_LINK, IntegrationUri = the internal listener ARN), with RequestParameters overwrite:header.Host = LoadBalancerHostName and TlsConfig.ServerNameToVerify set (see Q6);
exactly six routes and no catch-all: GET /ping, POST /github/ping, POST /github/push, POST /github/status, POST /watchtower/deploy and POST /cloudformation/deploy;
a $default stage with auto-deploy;
a regional AWS::ApiGatewayV2::DomainName for ApiHost, plus an ApiMapping and a Route 53 A alias record.

Mappings rule. The deploy action already passes ApiHost and LoadBalancerHostName, so those are declared as parameters. ApiPrefix stays undeclared, because the routes stay at root and the GitHub webhook URL does not change. Everything else new goes in the Deployment mapping: internal listener ARN, hosted zone, certificate ARN, VPC Link egress CIDR and the TLS server name. #427 later renames that mapping to Settings.

How the recent and planned work fits through the gateway:

feat(github): verify webhook signatures with the configured secret #426 (GitHub signatures): HTTP API proxy integrations forward the request body bytes and headers unmodified, so X-Hub-Signature-256, X-GitHub-Delivery and X-GitHub-Event reach the middleware as GitHub sent them. Coding Researcher will confirm this, and how X-Forwarded-For is populated (see b. and c.).
feat(cloudformation): Authenticate SNS deliveries to the cloudformation deploy endpoint #424 (SNS authentication): basic auth in the subscription URL arrives as Authorization, which a proxy integration with no authorizer passes through. SNS message-signature verification does not depend on transport. The SNS subscription URL moves to the API Gateway custom domain.
feat(watchtower): Authenticate Watchtower notifications to the watchtower deploy endpoint #425 (Watchtower authentication): custom headers and Authorization pass through. feat(watchtower): Authenticate Watchtower notifications to the watchtower deploy endpoint #425 must not use a header name API Gateway treats specially, such as X-Api-Key. Basic auth is the safe choice.
Not affected: /cloudformation/deploy keeps accepting SNS's text/plain body, and the ALB target-group health check (/ping) still hits the target group directly.

Test strategy

cfn-lint via the pre-commit hook. No C# changes, so the coverage ratchet skips this as a non-code change.
After deploy:
- redeliver a GitHub webhook through the gateway and confirm 204;
- send an unsigned POST to /github/push and confirm the app's 401 comes back (not an API Gateway 403/404);
- confirm /cloudformation/deploy with Content-Type: text/plain reaches the app;
- confirm a route not in the list (for example /foo) gets API Gateway's 404.

Assumptions

a. Payloads over 10 MB (the HTTP API request limit) are rejected by API Gateway with its own 413 before reaching the app. That is acceptable. #426's 25 MB BUFFER_LIMIT stays as it is.
b. API Gateway adds the client's source IP to X-Forwarded-For and the internal ALB appends to it, so #426's rejection log still shows the real client. The logged remote address remains the ALB's.
c. ApiHost and LoadBalancerHostName hold the same value per environment (both come from AWS_LOAD_BALANCER_HOSTNAME_DEV / _PROD).
d. Webhooks delivered during the DNS switch may be lost. GitHub does not automatically retry failed deliveries.

Open questions

Q4 to Q9 continue from the issue's original questions 1 to 3.

Q4. Which Route 53 hosted zone (ID or name) holds the dev and prod host names?
Q5. Is there an existing ACM certificate in eu-west-1 covering both host names (for example a wildcard)? Yes means give the ARN or ARNs; no means the stack requests one with DNS validation.
Q6. What name does the internal ALB's HTTPS listener certificate present, for the integration's ServerNameToVerify? Is it the same host name?
Q7. Is listener rule priority LoadBalancerPriority (20000 in prod) free on the internal listener? If not, which priority should be used?
Q8. The existing DNS records for the host names (currently pointing at the testing ALB) would make the Route 53 record creation fail. Should they be deleted by hand just before the deploy (yes), or imported into the stack (no)?
Q9. Sequencing: should #421 go before #427 (template format) and rebase over #428 (Discord webhook, in another session) once that lands? Both edit cloudformation.json.

This issue is marked Blocked under the plan-first approval gate (Issue Workflow: Plan First): implementation does not start until a human approves this plan and the open questions are answered.

Answers from the repository owner in live chat:

q4 funfair.io q5 yes arn:aws:acm:eu-west-1:117769150821:certificate/d9405a08-95c9-4924-acde-d86cda5f29e8 q6 buildbot.funfair.io q7 no use 50 q8 imported into stack q9 we should include 427 here given there's so many changes 428 will land shortly with any luck so will be able to rebase soon

Resolved:

Q4: hosted zone funfair.io.
Q5: use the existing certificate arn:aws:acm:eu-west-1:117769150821:certificate/d9405a08-95c9-4924-acde-d86cda5f29e8.
Q6: ServerNameToVerify is buildbot.funfair.io.
Q7: priority 20000 is not free on the internal listener; use 50.
Q8: import the existing DNS records into the stack.
Q9: refactor(cloudformation): Restore the shared parameter-based template format #427 (shared template format, with mappings except where the deploy workflow passes a parameter) is folded into this work. The work rebases over refactor(discord): replace Discord.Net bot login with a Discord webhook #428 once that lands.

A revised plan follows.

Implementation Plan

Revision of the plan above, incorporating the Q4 to Q9 answers and folding in #427.

Files to change

cloudformation.json:
1. restructure it into the shared format (refactor(cloudformation): Restore the shared parameter-based template format #427), using funfair-tech/funfair-ethereum-proxy-server as the reference;
2. move to the internal ALB;
3. add the API Gateway.
CHANGELOG.md: entries via dotnet changelog, including Deployment Changes for the DNS import and cut-over.
README.md: a deployment note that webhooks now arrive through API Gateway, plus the one-off DNS import steps.

Approach

Template format (#427), applying the owner's rule "everything should be in mappings unless its explicitly set as a parameter in the deploy workflow":

Parameters: only what .github/actions/cloudformation-deploy/action.yml passes and the template uses: ApplicationName, ApplicationContainer, ApplicationVersion, ApiHost, LoadBalancerHostName, LoadBalancerPriority and AwsContainerRegistry.
Settings / Values mapping: everything else, replacing the Deployment mapping. That covers:
- cluster name, task size, desired count and health percentages;
- container port and container owner (funfair);
- SecretsManagerArn, the execution role ARN (ecsTaskExecutionRole), SecurityGroupIDs, SubnetIDs and VpcID;
- the internal listener ARN, the hosted zone name (funfair.io.), the certificate ARN, the VPC Link egress CIDR (172.31.0.0/16) and the TLS server name (buildbot.funfair.io).
Image: ${AwsContainerRegistry}/${Owner}/${ApplicationContainer}:${ApplicationVersion}, as in the sibling templates.
Secrets: each entry uses the long-form Fn::Sub with a Fn::FindInMap variable for the ARN, because short-form Fn::Sub cannot read a mapping.
Kept from fix(cloudformation): inject ECS secrets via Secrets instead of plain Environment #420 / feat(github): verify webhook signatures with the configured secret #426: the ECS Secrets injection (including GITHUB__WEBHOOKSECRET) and the deployment circuit breaker.
Before merge: compare the resolved values against the current deployed stack. The image, execution role, secret ARN, cluster, task size, subnets and security groups must be unchanged.

Internal load balancer: ListenerRule moves to the internal listener. TargetGroup and the service's LoadBalancers block stay as they are. The rule priority is 50 (Q7; see Q10 for how to set it).

API Gateway:

an HTTP API with a stack-owned VPC Link, using the subnets from the mapping and a new security group with only egress TCP 443 to 172.31.0.0/16;
one HTTP_PROXY integration, with ConnectionType: VPC_LINK and IntegrationUri set to the internal listener ARN, plus:
- RequestParameters overwrite:header.Host set to LoadBalancerHostName;
- TlsConfig.ServerNameToVerify set to buildbot.funfair.io;
exactly six routes and no catch-all: GET /ping, POST /github/ping, POST /github/push, POST /github/status, POST /watchtower/deploy and POST /cloudformation/deploy;
a $default stage with auto-deploy;
a regional DomainName for ApiHost using the given ACM certificate, plus an ApiMapping;
a Route 53 A alias record in funfair.io. with DeletionPolicy: Retain. It is imported into the stack (Q8; see Q11).

How the recent and planned work fits through the gateway (unchanged from the previous plan):

feat(github): verify webhook signatures with the configured secret #426's signature check gets the body and headers unmodified.
feat(cloudformation): Authenticate SNS deliveries to the cloudformation deploy endpoint #424 (SNS) basic auth arrives as Authorization. SNS signature verification does not depend on transport, and the subscription URL moves to the custom domain.
feat(watchtower): Authenticate Watchtower notifications to the watchtower deploy endpoint #425 (Watchtower) uses basic auth and avoids X-Api-Key.

Test strategy

cfn-lint via the pre-commit hook. The coverage ratchet skips this as a non-code change.
Diff the resolved template values against the live stack: aws cloudformation get-template versus the rendered new template. This is run by the owner or with read-only AWS access.
After deploy:
- a redelivered GitHub webhook returns 204 through the gateway;
- an unsigned POST to /github/push returns the app's 401;
- /cloudformation/deploy with text/plain reaches the app;
- an unlisted route gets API Gateway's 404.

Assumptions

a. Payloads over 10 MB are rejected by API Gateway with its own 413 before reaching the app, and that is acceptable.
b. API Gateway puts the client source IP in X-Forwarded-For and the ALB appends to it. Coding Researcher confirms this during implementation.
c. ApiHost and LoadBalancerHostName hold the same value per environment.
d. Webhooks delivered during the DNS switch may be lost.
e. The certificate d9405a08-... covers the dev and prod host names.

Open questions

Q10. The deploy workflow passes LoadBalancerPriority (from AWS_LOAD_BALANCER_PRIORITY_DEV / _PROD), so under the mappings rule it stays a parameter. Should the priority of 50 be set by changing those repository variables (yes), or by hardcoding 50 in the Settings mapping and dropping the parameter (no)? What priority should dev use?
Q11. A Route 53 record import needs a one-off aws cloudformation create-change-set --change-set-type IMPORT, and the deploy action cannot do that. Will you run that import by hand from steps the PR documents, before the first deploy (yes)? Or should the cut-over avoid importing (no)?

This issue stays Blocked under the plan-first approval gate (Issue Workflow: Plan First) until a human approves this revised plan.

Coding Researcher

Questions:

How does an HTTP API HTTP_PROXY integration over a VPC Link handle the body and headers?
What does TlsConfig require against an HTTPS ALB listener?
Can the existing Route 53 record be imported into the stack?

Outcomes:

Body: passed through as-is. No mapping templates; the 100 KB truncation applies only to $request.body parameter mappings, which are not used. Payload limit 10 MB; header limit 10,240 bytes.
X-Forwarded-For: HTTP APIs turn incoming X-Forwarded-* into a standard Forwarded header. X-Forwarded-For is reserved and cannot be mapped. The ALB then creates X-Forwarded-For containing only the VPC Link's private IP, so feat(github): verify webhook signatures with the configured secret #426's rejection log would no longer show the real client. A non-reserved header can be set with $context.identity.sourceIp, for example overwrite:header.X-Client-Ip. Sources: known issues, parameter mapping, ALB X-Forwarded headers.
Host override: allowed (not reserved).
Stage prefix: the $default stage adds no path prefix.
TlsConfig: required. Without it, private integrations use plain HTTP. ServerNameToVerify must match a CN/SAN on the ALB certificate and is sent as SNI. An ACM public certificate is fine. Private-CA and self-signed certificates are not usable. Source: private integrations.
VPC Links: immutable (a change replaces them). They go INACTIVE after 60 days without traffic. Subnet AZs must be on the supported list.
Route 53 import: Not possible. AWS::Route53::RecordSet is not in CloudFormation's resource-import support list, and a normal deploy cannot adopt an existing record (it fails with "already exists"). Source: resource import support. Alternatives:
- (a) keep the record outside the stack and switch its alias with a Route 53 UPSERT after the deploy (atomic, no gap);
- (b) delete the record and let the stack create it (an NXDOMAIN window of up to about 15 minutes);
- (c) use a new stack-owned host name and change the webhook URLs.

Answer from the repository owner in live chat, after research showed that a Route 53 record set cannot be imported:

q16 b

Resolved, superseding Q8/Q11: the existing buildbot.funfair.io record is deleted by hand just before the first deploy of #433, and the stack then creates and owns the AWS::Route53::RecordSet (A alias to the API Gateway custom domain). Expect an NXDOMAIN window of up to about 15 minutes while resolver negative caches expire. The README and the Deployment Changes entry will document the ordering: delete the record, then deploy immediately. No import procedure is needed.

Answer from the repository owner in live chat:

q17 yes

Resolved: the API Gateway integration sets overwrite:header.X-Client-Ip = $context.identity.sourceIp. Because it overwrites, a client-supplied value cannot spoof it. As its own change in #433, the #426 webhook rejection log adds that header (sanitised like the other client values), so rejections behind the gateway still show the real caller.

feat(cloudformation): front BuildBot with an API Gateway HTTP API ove…

feat(github): log the API Gateway client IP on rejected webhooks (#421)

docs(readme): cover recovery from a rolled-back first deploy (#421)

docs(changelog): describe API Gateway, internal ALB and template form…

feat(cloudformation): throttle the API Gateway stage and log access (#…

feat(cloudformation): keep ECS application logs for 30 days (#421)

docs(changelog): note 30-day ECS log retention (#421)

docs(changelog): describe the ECS service naming by its effect (#421)

fix(cloudformation): make the task definition depend on its log group (…

docs(changelog): note the log group ordering fix (#421)

docs(coverage): record coverage at 5838d6a (#421)

Navigation Menu

Repository navigation

Move to the internal load balancer behind an API Gateway #421

Summary

Changes

1. Switch to the internal load balancer

2. Add an API Gateway that forwards to the internal load balancer

Notes / open questions

Implementation Plan

Files to change

Approach

Test strategy

Assumptions

Open questions

Implementation Plan

Files to change

Approach

Test strategy

Assumptions

Open questions

5 remaining items

Coding Researcher

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Move to the internal load balancer behind an API Gateway #421

Description

Summary

Changes

1. Switch to the internal load balancer

2. Add an API Gateway that forwards to the internal load balancer

Notes / open questions

Activity

credfeto commented on Oct 13, 2026

credfeto commented on Oct 13, 2026

credfeto commented on Oct 13, 2026

credfeto commented on Oct 13, 2026

credfeto commented on Oct 15, 2026

credfeto commented on Oct 15, 2026

credfeto commented on Oct 15, 2026

credfeto commented on Oct 15, 2026

Implementation Plan

Files to change

Approach

Test strategy

Assumptions

Open questions

credfeto commented on Oct 15, 2026

credfeto commented on Oct 15, 2026

Implementation Plan

Files to change

Approach

Test strategy

Assumptions

Open questions

5 remaining items

credfeto commented on Oct 16, 2026

Coding Researcher

credfeto commented on Oct 16, 2026

credfeto commented on Oct 16, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions