aws-nuke-exporter doesn't report lines lacking ID or complete Details

[dupuy]

The sample_output folder demonstrates this for two of the three cases:

1. Missing Details - line in text output with only three - is omitted:

aws-nuke only prints details when it is possible to filter based on properties. Not all resource types support this.

% grep AppStreamImage sample_output/sample_nuke_output.txt 
xxx-xxx-xxx - AppStreamImage - xxx-xxx-xxx - cannot delete public AWS images
% grep -l AppStreamImage sample_output/*
sample_output/sample_nuke_output.txt
%

2. Incomplete Detail - line in text output missing closing ] is omitted (scroll right to see <<OutputTruncated>>):

Apparently, sometimes the Details data gets too long and is truncated? I haven't seen this, but it is in your samples.

% grep terraform sample_output/sample_nuke_output.txt 
global - IAMRolePolicy - xxx-xxx-xxx -> terraform-2023111309155894560000000c - [PolicyName: "terraform-2023111309155894560000000c", role:CreateDate:<<OutputTruncated>>
% grep -l terraform sample_output/*
sample_output/sample_nuke_output.txt
%

3. Detail present but missing ID

Some resource types don't have an ID that can be used for filtering, and aws-nuke can only filter with properties. For these, aws-nuke doesn't print an ID, but just skips ahead to Details.

us-east-1 - ECSTask - [ClusterARN: "arn:aws:ecs:us-east-1:xxxxxxxxxxxx:cluster/zzz", TaskARN: "arn:aws:ecs:us-east-1:xxxxxxxxxxxx:task/xxxxxx178008dbb84d49e7b7ade6ff37dccba5"] - would remove

The above cases should be sufficient for test purposes, but I can provide some more (obfuscated) examples.

An entry of particular interest is the one for the resource type NetpuneSnapshot (sic). I don't know if this is an aws-nuke error or a bug in the boto code, or in the AWS API itself, nor whether you can use that type for filters (and if you can, whether the filter works correctly). This would bear further investigation, but I dont have time right now.

us-east-1 - NetpuneSnapshot - rds:aurora-cluster-demo-2024-03-xx-xx-xx - would remove

Here is a more extensive list

us-east-1 - ECSCluster - arn:aws:ecs:us-east-1:xxxxxxxxxxxxx:cluster/xxxx-xxxxx - would remove
us-east-1 - SFNStateMachine - arn:aws:states:us-east-1:xxxxxxxxxxxxx:stateMachine:SimpleAsyncWorkflow462ECA3D-WfikQto27RjB - would remove
us-east-1 - CloudWatchEventsTarget - Rule: AWSControlTowerManagedRule Target ID: ControlTower-ManagedRuleTarget - filtered by config
us-east-1 - CloudWatchEventsTarget - Rule: AutoScalingManagedRule Target ID: autoscaling - would remove
us-east-1 - GlueDatabase - default - would remove
us-east-1 - CognitoUserPoolDomain - CognitoPool -> dns-name-7im7l1ccahjb9df9dmhvntu - would remove
us-east-1 - ResourceGroupGroup - AppManager-CFN-CDKToolkit - would remove
us-east-1 - ResourceGroupGroup - AppManager-CFN-StackSet-AWS-QuickSetup-SSMHostMgmt-LA-83a0h-a6a85309-14a4-4945-8c8c-a4dbc6a19e57 - would remove
us-east-1 - SageMakerNotebookInstance - BasicNotebookInstance-nTWO30HDG - would remove
us-east-1 - ConfigServiceConfigurationRecorder - aws-controltower-BaselineConfigRecorder - filtered by config
us-east-1 - SSMDocument - AWSQuickSetup-CreateAndAttachIAMToInstance-80h - would remove
us-east-1 - AppStreamImage - AppStream-Graphics-Design-WinServer2019-01-26-2024 - cannot delete public AWS images
us-east-1 - AWSBackupVaultAccessPolicy - aws/efs/automatic-backup-vault - would remove
us-east-1 - GlueCrawler - mac-training-crawler - would remove
us-east-1 - ECSTaskDefinition - arn:aws:ecs:us-east-1:xxxxxxxxxxxxx:task-definition/ecs-cloud-xxx-agent:1 - would remove
us-east-1 - ECSTaskDefinition - arn:aws:ecs:us-east-1:xxxxxxxxxxxxx:task-definition/ecs-cloud-linux-ec2:1 - would remove
us-east-1 - ECSTaskDefinition - arn:aws:ecs:us-east-1:xxxxxxxxxxxxx:task-definition/ecs-cloud-linux-fargate:1 - would remove
us-east-1 - GlueJob - mac-training-etl-job - would remove
us-east-1 - NeptuneInstance - tf-202308xxxxxxx32000000003 - would remove
us-east-1 - SQSQueue - https://sqs.us-east-1.amazonaws.com/xxxxxxxxxxxx/xxxx-test-q - would remove
us-east-1 - MediaConvertQueue - Default - cannot delete default queue
us-east-1 - SSMAssociation - 09xxxxe8-xxxx-xxxx-90c0-fe95xxxx08e9 - would remove
us-east-1 - AppConfigDeploymentStrategy - [ID: "AppConfig.AllAtOnce", Name: "AppConfig.AllAtOnce"] - cannot delete predefined Deployment Strategy
us-east-1 - CognitoUserPool - CognitoPool - would remove
us-east-1 - ServiceDiscoveryService - srv-npyxxxx5yuekl3 - would remove
us-east-1 - SageMakerNotebookInstanceState - BasicNotebookInstance-nTWO3ElT0HDG - would remove
us-east-1 - CloudWatchEventsRule - Rule: AWSControlTowerManagedRule - filtered by config
us-east-1 - CloudWatchEventsRule - Rule: AutoScalingManagedRule - would remove
us-east-1 - GlueTrigger - start - would remove
us-east-1 - ECSService - arn:aws:ecs:us-east-1:xxxxxxxxxxxx:service/jmaster -> arn:aws:ecs:us-east-1:xxxxxxxxxxxxx:cluster/zzz - would remove
us-east-1 - ECSTask - [ClusterARN: "arn:aws:ecs:us-east-1:xxxxxxxxxxxx:cluster/zzz", TaskARN: "arn:aws:ecs:us-east-1:xxxxxxxxxxxx:task/xxxxxx178008dbb84d49e7b7ade6ff37dccba5"] - would remove
us-east-1 - ServiceDiscoveryNamespace - ns-gxd7viamvwu - would remove
us-east-1 - NetpuneSnapshot - rds:aurora-cluster-demo-2024-03-xx-xx-xx - would remove
us-east-1 - ConfigServiceDeliveryChannel - aws-controltower-BaselineConfigDeliveryChannel - filtered by config
us-east-1 - SNSSubscription - Owner: xxxxxxxxxxxxx ARN: arn:aws:sns:us-east-1:xxxxxxxxxxxxx:aws-controltower-SecurityNotifications:991f621a-39f5-4294-9d3e-76da1a4e1845 - filtered by config
us-east-1 - SNSSubscription - Owner: xxxxxxxxxxxxx ARN: arn:aws:sns:us-east-1:275279264324:mac-re-AwsHealthNotification-LogError-Topic:0cxxxx56-404a-488a-9574-b9xxxx57cc49 - would remove
us-east-1 - SNSSubscription - Owner: xxxxxxxxxxxxx ARN: arn:aws:sns:us-east-1:275279264324:SimpleAsyncWorkflow-TextractAsyncTextractAsyncSNSBB89DC08-Cqgsu9I4Lu6l:111d6e8e-c74d-4e70-8302-c32ba2a734a1 - would remove
us-east-1 - NeptuneCluster - aurora-cluster-demo - would remove
us-east-1 - ServiceDiscoveryInstance - 178008dbb84d49e7b7ade6ff37dccba5 -> srv-npytkwz265yuekl3 - would remove
us-east-1 - LifecycleHook - Launch-LC-Hook - would remove
us-east-1 - LifecycleHook - Terminate-LC-Hook - would remove
us-east-1 - LifecycleHook - Launch-LC-Hook - would remove
us-east-1 - LifecycleHook - Terminate-LC-Hook - would remove
us-east-1 - OpsWorksUserProfile - arn:aws:sts::xxxxxxxxxxxxx:assumed-role/AWSReservedSSO_zzzzzzz/[email protected] - Cannot delete OpsWorksUserProfile of calling User
us-east-2 - ECSCluster - arn:aws:ecs:us-east-2:xxxxxxxxxxxxx:cluster/django_test - would remove
us-east-2 - CloudWatchEventsTarget - Rule: AWSControlTowerManagedRule Target ID: ControlTower-ManagedRuleTarget - filtered by config
us-east-2 - CloudWatchEventsTarget - Rule: aws-controltower-ConfigComplianceChangeEventRule Target ID: Compliance-Change-Topic - filtered by config
us-east-2 - CloudWatchEventsTarget - Rule: security-notify Target ID: Id5f453d2f-d5af-4c16-adbc-2dxxxff008 - would remove
us-east-2 - CognitoUserPoolDomain - xxxx0285_userpool_3bc85-dev -> xxxxx0285-3bc85-dev - would remove
us-east-2 - ConfigServiceConfigurationRecorder - aws-controltower-BaselineConfigRecorder - filtered by config
us-east-2 - AppStreamImage - AppStream-Graphics-Design-WinServer2019-01-26-2024 - cannot delete public AWS images
global - IAMSAMLProvider - arn:aws:iam::xxxxxxxxxxxxx:saml-provider/AWSSSO_e5xxxxxdec00ecbc_DO_NOT_DELETE - filtered by config

[oijkn]

Thank you for reaching out with your concerns and observations regarding the aws-nuke-exporter tool. I'd like to address each of your points as follows:

1. Export Format Issue (Fixed): I'm glad to inform you that the first issue regarding the export format has been corrected. Thanks for bringing this to our attention.

2. Incomplete Detail and Truncation Concerns: Regarding your second point, it seems you're encountering issues with lines in the aws-nuke output that have missing closing brackets or are marked with <<OutputTruncated>>. This typically indicates that the details data is too long and gets truncated in the output. To better assist you, could you please provide more specific examples or clarify the exact nature of the problem you're facing? This will help us understand the issue in depth and explore potential solutions.

3. Additional Examples and Handling Specific Cases: For the third point, I've added examples in the sample_output/sample_nuke_output.txt file to demonstrate how the tool handles resources like ECSTask and NeptuneSnapshot, including cases where the ID is missing, or the details are incomplete. These examples should help clarify how the exporter processes various scenarios and ensures that the output remains informative and structured.

Regarding the resource type NetpuneSnapshot (possibly a typo for NeptuneSnapshot), I acknowledge your concern about whether it's an error from aws-nuke, a bug in the boto code, or an issue with the AWS API itself. This is certainly something worth investigating further. However, as I'm currently limited in time, I encourage the community or other contributors to look into this peculiar case and share any findings or insights.

Your feedback is invaluable in helping us enhance the tool's functionality and reliability. Please feel free to provide any additional information or examples regarding the second point, and I'll do my best to address it promptly.

Thank you for your contribution to making aws-nuke-exporter better.

[dupuy]

Thanks for your very prompt response and the 1.0.3 release, which resolved the problems in two of the cases. However, it didn't correctly handle the missing ID or truncated Detail cases correctly, placing those into RemovalStatus:

% grep 'RemovalStatus": "[^cw]' sample_output/sample_nuke_output.json
                "RemovalStatus": "[PolicyArn: \"xxx-xxx-xxx\", PolicyName: \"MigrationHubServiceRolePolicy\", RoleCreateDate: \"xxx-xxx-xxx\", RoleLastUsed: \"xxx-xxx-xxx\", RoleName: \"AWSServiceRoleForMigrationHub\", RolePath: \"/xxx-xxx-xxx - cannot detach from service roles"
                "RemovalStatus": "[PolicyName: \"terraform-2023111309155894560000000c\", role:CreateDate:<<OutputTruncated>>"
                "RemovalStatus": "arn:aws:ecs:us-east-1:xxxx:cluster/ecs-cluster-xxxx - [ClusterARN: \"arn:aws:ecs:us-east-1:xxxx:cluster/ecs-cluster-xxxx\", TaskARN: \"arn:aws:ecs:us-east-1:xxxx:task/ecs-cluster-xxxx/xxxxxxxxxxxx\"] - would remove"

Also, the sample TXT output for the missing ID case had an ID with ->, rather than no ID at all, which is the output I get from aws-nuke 2.25.0, the most recent release (from last August). I don't know if there were earlier releases with different behavior.

Anyhow, I came up with PR #6, which handles all the sample cases pretty reasonably (there's no perfect solution for truncated output). Unfortunately you can't use the same name for multiple match groups, which complicates the code a bit, but it's not too painful.

[dupuy]

A quick google search turned up the reason for the typo NetpuneSnapshot: rebuy-de/aws-nuke#1108 and a fix was made last September (but there has been no release since last August).