chore(tests): add LMI e2e suite and run-scoped shared capacity provider - #5465
chore(tests): add LMI e2e suite and run-scoped shared capacity provider#5465svozza wants to merge 18 commits into
Conversation
powertools-for-aws-oss-automation
Bot
added
the
size/XL
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
powertools-for-aws-oss-automation
Bot
added
size/XL
|
… Managed Instances Adds an opt-in (RUN_LMI_TESTS) e2e suite that provisions an ephemeral Lambda Managed Instances capacity provider (dual-stack IPv6 VPC, no NAT, CloudWatch Logs interface endpoint) and proves the InvokeStore-backed log attribute isolation across invocations multiplexed into the same execution environment via a module-scoped promise barrier. Testing utils changes: - TestLmiCapacityProvider construct + ./resources/capacity-provider export - ExtraTestProps.lmi option on TestNodejsFunction (memorySize>=2048 floor, $LATEST.PUBLISHED qualified output) - includeTailLogs opt-out on invoke helpers (Tail unsupported on LMI) Related to #5092, findings posted on the issue.
… CloudWatch polling The handler tees process.stdout.write (the Logger's production write path) and returns its own log lines in the response payload, making log collection deterministic — no FilterLogEvents polling, no ingestion latency, no CloudWatch client in the test. Captured lines are filtered by the InvokeStore-scoped invocationKey rather than function_request_id: addContext stores the Lambda context in instance state, so under LMI multiplexing the request id stamped on log lines can belong to a different invocation. Tracked separately as a bug (see lmi-request-id-bug-handoff.md); once fixed, the filter should flip back to function_request_id as a regression check.
…ogs by request id With the lambda context now scoped per invocation under LMI (#5430), the handler selects its own log lines by function_request_id — an independent per-invocation attribute — and the test asserts the InvokeStore-scoped invocationKey on those lines, making the suite a regression check for both the context scoping and appendKeys isolation. The suite adds ~4 minutes to the logger e2e cell, inside the agreed budget for running on every e2e dispatch, so the RUN_LMI_TESTS gate and workflow input are removed.
The e2e suites are I/O-bound (waiting on CloudFormation), but vitest sizes its worker pool from CPU cores, so on 2-core CI runners only 3 of the 6 logger e2e files ran concurrently and the rest queued — each then paying its own stack deploy after waiting. With 8 workers all suites deploy their stacks up front and the cell duration approaches the slowest single suite (~6 min measured) instead of a serialized ~8 min. CloudFormation read-API pressure from the extra concurrent stack monitors is bounded by the existing DescribeStackEvents polling patch in the testing package (10s interval per stack).
With all 36 matrix cells sharing one account, the extra concurrent stack operations from 8-worker logger cells pushed the account-wide CloudFormation API rate over the edge: three unrelated cells failed with 'Throttling: Rate exceeded' on stack deploys. Back to default worker sizing; cell-duration work moves to the run-scoped shared capacity provider follow-up, which removes per-cell VPC/CP stacks entirely instead of racing them.
Instead of every LMI suite provisioning its own capacity provider + VPC, a workflow run can deploy one shared capacity provider per architecture up front and pass its ARN to each suite via LMI_CAPACITY_PROVIDER_ARN. - add packages/testing/src/lmi/cli.ts with deploy/destroy commands that manage a run-scoped LmiShared-<runId>-<arch> stack and print the capacity provider ARN on stdout - TestNodejsFunction accepts a capacity provider ARN string and attaches the function via L1 CfnFunction.capacityProviderConfig (an imported capacity provider has no addFunction) - ExtraTestProps.lmi.capacityProvider widened to CapacityProvider | string - TestLmiCapacityProvider ctor loosened to Pick<TestStack, 'stack'> so the CLI can build the stack standalone - logger lmi.test.ts reads LMI_CAPACITY_PROVIDER_ARN, falling back to an ephemeral per-suite capacity provider when unset
The --run-id argument flows into a CloudFormation stack name and, via join(tmpdir(), ...), into the assembly output path that is later read back. A crafted value (e.g. containing ../) could escape tmpdir(), so restrict it to the alphanumerics-and-hyphens set CloudFormation already requires for stack names and reject anything else at the boundary.
The path-injection scanner's taint analysis doesn't recognise the run-id regex guard as sanitisation, and its guidance is to validate the constructed path before touching the file system. Resolve the assembly output directory and assert it stays within os.tmpdir() before it is written to or read back, keeping the run-id validation as the input-side guard for defence in depth.
The constructor's cognitive complexity hit 17 (limit 15) once it carried the nested ARN-vs-construct capacity-provider branching. Move that logic into two private methods so the constructor keeps a flat structure and the two attachment strategies read independently. No behaviour change.
…hared capacity provider
dreamorosi
left a comment
dreamorosi
left a comment
There was a problem hiding this comment.
These are findings from a deep review pass, grouped as 2 blockers plus several smaller fixes. Happy to discuss any of them.
| * resolve the ARN from the stack outputs instead (see the e2e workflow). | ||
| */ | ||
| const main = async (): Promise<void> => { | ||
| await Promise.all( |
There was a problem hiding this comment.
Blocker: Running two TestStack.deploy() calls concurrently in one process also runs two fromAssemblyBuilder synths concurrently. toolkit-lib explicitly does not support this unless clobberEnv: false: each synth temporarily replaces the global process.env with an immutable proxy via temporarilyWriteEnv, and interleaved disposal can leave the process with the wrong/stale env (including leaked CDK_OUTDIR/CDK_CONTEXT_JSON). This works today only because each App is constructed before the synth window, so it is correct by accident. Could we either pass clobberEnv: false in TestStack.#synthAssembly() (safe here because the builder ignores the injected env), or deploy the architectures sequentially?
| * teardown from being attempted. | ||
| */ | ||
| const main = async (): Promise<void> => { | ||
| const results = await Promise.allSettled( |
There was a problem hiding this comment.
The same concurrent-synth concern described in deploySharedCapacityProvider.ts applies here.
| # "re-run all jobs": this teardown destroys the stacks at the end of each | ||
| # attempt, so a re-run LMI cell fails fast at ARN resolution ("stack does | ||
| # not exist") until the setup job re-runs and redeploys them. | ||
| teardown-lmi-capacity-providers: |
There was a problem hiding this comment.
Blocker: If an LMI cell is cancelled or times out before its afterAll runs, the per-suite function stack survives with its function still attached to the shared capacity provider. This teardown then tries to delete the provider stack while associations/ENIs still exist—the classic DELETE_FAILED path—leaking an EC2-backed fleet plus VPC with no sweeper to catch it. Could we add at least a retry/second pass in the destroy script, and/or scheduled cleanup for LmiShared-* stacks and stacks tagged Service: Powertools-for-AWS-e2e-tests?
| # suites and are subject to account vCPU quotas, so the suites share one per | ||
| # architecture instead of provisioning their own (see | ||
| # packages/testing/src/lmi/sharedCapacityProviderStack.ts). | ||
| setup-lmi-capacity-providers: |
There was a problem hiding this comment.
Related to the teardown concern: none of the three new jobs sets timeout-minutes, so the 360-minute default lets a hung deploy/cell keep two EC2 fleets alive for up to six hours before teardown even starts. The suite hooks themselves cap at 20 minutes each, so a roughly 60-minute job timeout seems easy to justify.
| arn=$(aws cloudformation describe-stacks \ | ||
| --stack-name "LmiShared-${GITHUB_RUN_ID}-${ARCH//_/-}" \ | ||
| --query "Stacks[0].Outputs[?OutputKey=='CapacityProviderArn'].OutputValue" \ | ||
| --output text) |
There was a problem hiding this comment.
Should-fix: --output text exits 0 with an empty string when the JMESPath filter matches nothing, and prints None when outputs are null. An empty value is silently treated as “no shared provider” by the test, so all four cells fall back to provisioning their own VPC and capacity provider, defeating the shared-provider design and pressuring the VPC quota without warning. Please fail unless the value is non-empty and not None, for example: [[ -n "$arn" && "$arn" != "None" ]] || exit 1.
| minExecutionEnvironments, | ||
| maxExecutionEnvironments, | ||
| } = scaling; | ||
| capacityProvider.addFunction(this, { |
There was a problem hiding this comment.
Should-fix: The ARN path explicitly sets publishToLatestPublished = true at line 131, while this construct path relies on CDK's @default - True even though the underlying CFN property documents no default. Passing publishToLatestPublished: true here would make both attach paths emit identical templates and remove dependence on an undocumented service default. This is also the path used by every local run and the CI fallback.
| "devDependencies": { | ||
| "@aws-lambda-powertools/testing-utils": "file:../testing" | ||
| "@aws-lambda-powertools/testing-utils": "file:../testing", | ||
| "@aws-sdk/client-cloudwatch-logs": "^3.1079.0", |
There was a problem hiding this comment.
Should-fix: This dependency is not imported anywhere in the package and looks left over from the CloudWatch-polling approach, so it can be dropped. Conversely, @aws-sdk/client-lambda is imported by tests/e2e/lmi.test.ts but is not declared here (it currently resolves via workspace hoisting), so that dependency should be declared instead.
|
|
||
| describe('Logger E2E - Lambda Managed Instances', () => { | ||
| // The LMI scheduler scales out to fresh execution environments until the | ||
| // capacity provider's fleet is saturated (8 environments with a 12 vCPU |
There was a problem hiding this comment.
Should-fix (comment accuracy): The real CI run (29740823390) reported 30/30 responses; 12 multiplexed across 24 execution environments in all four cells, so the “fleet saturates at 8 environments” model here—and in the similar file-header comment—does not match observed behavior, likely because both runtime cells now share each per-architecture provider. The assertion only needs one overlap, so the test held, but could we describe the actual mechanics so future readers do not tune counts against the wrong model?
| const invokeFunctionOnce = async ({ | ||
| functionName, | ||
| payload = {}, | ||
| includeTailLogs = true, |
There was a problem hiding this comment.
Should-fix: This new option has no callers; the LMI suite hand-rolls its own LambdaClient/InvokeCommand instead. Could we either wire the suite through invokeFunction or drop the option? Also, with includeTailLogs: false, callers currently receive silently empty TestInvocationLogs rather than an error.
dreamorosi
left a comment
dreamorosi
left a comment
There was a problem hiding this comment.
Apologies for the long wait - I just finally found some time to review this now.
Some review comments that I think are valid, let me know if you disagree and/or I misunderstood anything.
|
The infrastructure-leak concern raised in review—cancelled cells leaving functions attached to the shared capacity provider and causing |
3 participants
Summary
Adds an end-to-end test that validates Logger's
InvokeStore-backed log-attribute isolation on Lambda Managed Instances (LMI), where multiple invocations run concurrently in the same execution environment. Also introduces the testing infrastructure to support it, including a run-scoped shared capacity provider so LMI suites don't each provision their own VPC + capacity provider.Changes
packages/logger/tests/e2e/lmi.test.ts) that saturates a deliberately small fleet, forces concurrent invocations to multiplex into shared execution environments, and asserts each invocation's appended keys stay isolatedprocess.stdout.write(the production write path) and returning them in the response payload, making log collection deterministic — no CloudWatch polling or ingestion latencyInvokeStore-scopedinvocationKey, and additionally assert onfunction_request_idas a regression check now that the Lambda context is scoped per invocation under LMI (fix(logger): scope lambda context per invocation under LMI concurrency #5430)TestLmiCapacityProvider(dual-stack IPv6-only VPC, egress-only IGW, CloudWatch Logs interface endpoint — no NAT) and extendTestNodejsFunctionto attach a function to a capacity provider by construct or by ARNpackages/testing/src/lmi/cli.tswith deploy/destroy commands that manage a run-scopedLmiShared-<runId>-<arch>stack and print the capacity provider ARN, so a workflow run can provision one shared capacity provider per architecture up front and pass it to each suite viaLMI_CAPACITY_PROVIDER_ARN(suites fall back to an ephemeral per-suite provider when unset)RUN_LMI_TESTSgate and workflow input)Issue number: closes #5518
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.