`newrelic-lambda integrations update` fails to update pre-0.9 stack due to `No newrelic-log-ingestion function in region ...` error

[huonw]

Description

Steps to Reproduce

1. Create a stack with the "old" function name, i.e. using a CLI version 0.8.0 or earlier
2. Attempt to update it with newrelic-lambda integrations update ... using CLI version 0.9.0

Expected Behaviour

The update should succeed, either renaming the function or preserving it with the old name.

It looks like #254 may have introduced some if statements that aren't working as intended given its stated goal of preserving compatibility.

newrelic-lambda-cli/newrelic_lambda_cli/integrations.py

Lines 89 to 103 in f472e28

	def get_unique_newrelic_log_ingestion_name(session, stackname=None):
	if not stackname:
	stackname = INGEST_STACK_NAME
	stack_id = _get_cf_stack_id(session, stack_name=stackname)
	if stack_id:
	return "newrelic-log-ingestion-%s" % (stack_id.split("/")[2].split("-")[4])
	def get_newrelic_log_ingestion_function(session, stackname=None):
	unique_log_ingestion_name = get_unique_newrelic_log_ingestion_name(
	session, stackname
	)
	if unique_log_ingestion_name:
	function = get_function(session, unique_log_ingestion_name)
	return function

In addition, it would be nice if the logs were more specific about which function doesn't exist. I would've identified the problem with far less effort if the error printed out No newrelic-log-ingestion-0aa... function in region, given we do have function called newrelic-log-ingestion as literally appears in the error message.

Relevant Logs / Console output

The update fails, with error:

✖️ No newrelic-log-ingestion function in region 'ap-southeast-2'. Run 'newrelic-lambda integrations install' to install it.
✖️ Update Incomplete. See messages above for details.

I inserted debug logs like the following:

modified   newrelic_lambda_cli/integrations.py
@@ -98,6 +98,7 @@ def get_newrelic_log_ingestion_function(session, stackname=None):
     unique_log_ingestion_name = get_unique_newrelic_log_ingestion_name(
         session, stackname
     )
+    print(f"DEBUG: {unique_log_ingestion_name=}")
     if unique_log_ingestion_name:
         function = get_function(session, unique_log_ingestion_name)
         return function

It prints DEBUG: unique_log_ingestion_name='newrelic-log-ingestion-0aa...'. That is, the function name has a unique suffix. Our AWS account indeed doesn't have a lambda with this name; although does have newrelic-log-ingestion.

Your Environment

macOS

# requirements.txt
newrelic-lambda-cli==0.9.0
boto3==1.34.59
botocore==1.34.59
jmespath==1.0.1
python-dateutil==2.9.0.post0
six==1.16.0
urllib3==2.0.7
s3transfer==0.10.0
click==8.1.7
colorama==0.4.6
emoji==2.10.1
gql==2.0.0
graphql-core==2.3.2
promise==2.3
Rx==1.6.3
requests==2.31.0
charset-normalizer==3.3.2
idna==3.6
certifi==2024.2.2
tabulate==0.9.0

Additional context

N/A

[huonw]

Oh. Maybe this is intended behaviour, according to https://github.com/newrelic/newrelic-lambda-cli/releases/tag/v0.9.0.

If that's the case, how do we update our stacks? Are we stuck on 0.8.0 forever? Maybe there's some method to manually update them, or maybe this script could be upgraded to support the migration automatically?

[chaudharysaket]

Hi @huonw, the new release assumes that one would create a new log-ingestion function with cli version 0.9 and use it for integration.

newrelic-lambda integrations install \
    --nr-account-id <account id> \
    --nr-api-key <api key> \
    --stackname <newrelic-primary-log-ingestion> # sample name for stackname 

The above command would create a new stack with a new log-ingestion-function with some suffix and using same stackname you can subscribe your lambdas.

newrelic-lambda subscriptions install --function <name or arn> --stackname <newrelic-primary-log-ingestion>

Yes, as per the release notes, this change is not compatible with log-ingestion function created with earlier versions. Also could you please indicate what you want to update? Is it the log-ingestion function? Thanks

[huonw]

Thanks for the reply.

We have a stacks we created with newrelic-lambda integrations install --nr-account-id <account id> --nr-api-key <api key> with an old version (by the dates, maybe with v0.7.2 but I don't know for sure). We want to upgrade those stack to the latest version using newrelic-lambda integrations update ..., preferably without any service interruption. (We want to pick up the fix for #248 specifically.)

I tried using 0.9.0 and was getting the errors described in the issue and didn't know the cause, until diving into the code. I'm still somewhat unclear on what the recommended upgrade path is: delete the integration stack and recreate it?

(As a stop gap, I've just used 0.8.0 and that's been enough to resolve the immediate problem.)

As a user, it would've helped me to either have:

1. the 0.9.0 upgrade work without issues
2. if the compatibility break is required, more information about how to upgrade: preferably both of:
   • guidance in the 0.9.0 release notes, it currently says it's unsupported but it could also provide a recommendation for how to resolve the issue
   • the error message itself reference this (e.g. ✖️ No newrelic-log-ingestion-0ab... function in region 'ap-southeast-2'. If you have an stack created with an old version, follow instructions in <link to release notes> to uopgrade; if not, run 'newrelic-lambda integrations install' to install it.)

[rittneje]

We also encountered this issue. As @huonw mentioned, the error message is completely useless. I'm also very disappointed that it's been almost three months without any solution from New Relic.

I think the decision in v0.9.0 needs to be reversed until New Relic has a valid plan for migrating to the new function name. Also, I will say that adding a suffix to the function seems totally unnecessary. If anything, you should force the other thing to change to get rid of the suffix.

[chaudharysaket]

We are looking into the ways to make the CLI changes backward compatible. Appreciate the feedback, Thank you!

[rittneje]

@chaudharysaket is there any update on this?

[rittneje]

For anyone stuck on this upgrade, here are the steps we followed.

1. Download the stack template for the existing NewRelicLogIngestion stack.
2. Look at the parameter values for the NewRelicLogIngestion stack. For the NRLicenseKey parameter in particular, you may extract its value from the LICENSE_KEY environment variable of the newrelic-log-ingestion Lambda function.
3. Delete the NewRelicLogIngestion stack. Use the template you downloaded earlier to re-create it under a new name (e.g., NewRelicLogIngestion-OLD), with the same parameter values.
4. Using CloudShell, install the latest CLI via pip3 install --upgrade newrelic-lambda-cli
5. Install the new stack via newrelic-lambda integrations install --nr-account-id '...' --nr-api-key 'NRAK-...' --linked-account-name '...' --enable-cw-ingest --disable-license-key-secret For the linked account name, you can reuse your current name from New Relic under Infrastructure > AWS. For the API key, you just need an arbitrary user key from the account is question, which is only used to find the license key at install time. You may delete the user key after the installation.
6. Migrate all subscriptions. Due to a bug in the CLI (see newrelic-lambda subscriptions install does not consider destination arn of existing subscription filter #357), you have to run this command twice, once with a bogus filter and once without. So first do newrelic-lambda subscriptions install --function installed --filter-pattern XXX (that's literally XXX), and then do newrelic-lambda subscriptions install --function installed
7. Once you have confirmed the old newrelic-log-ingestion is completely unused, you may delete the NewRelicLogIngestion-OLD stack you created in step 3.