AWSTemplateFormatVersion: '2010-09-09' Description: >- Netlumi Security Monitoring: a read-only scanner role, EventBridge forwarding rules for real-time changes, and optionally a CloudTrail trail. Metadata: # This template's version. Netlumi compares it with the newest to tell you when an update is available. NetlumiTemplateVersion: '23' AWS::CloudFormation::Interface: ParameterGroups: - Label: default: Set by Netlumi (do not change) Parameters: - RegistrationId - ExternalId - CallbackSecret - CallbackUrl - NetlumiAccountId - Label: default: Your choices Parameters: - AccountType - Regions - RealtimeForwarding - EnableCloudTrail - ForwardAIInvocationEvents - AgentMonitoring - AgentMonitoringMonthlyCapUSD - EnableDnsQueryLogging - DnsQueryLoggingVpcIds - EksClusters - OrgBootstrapAccess - ExtraStateObjectArns - StateBuckets - StateLockDiscovery - PermissionsBoundaryArn Parameters: RegistrationId: Type: String Description: Netlumi registration this stack completes. Set by Netlumi. AllowedPattern: '^reg-[0-9a-f]{32}$' ExternalId: Type: String NoEcho: true Description: >- Secret ExternalId Netlumi presents when it assumes the scanner role. Unique to this connection. Set by Netlumi; do not share. MinLength: 32 MaxLength: 1224 AllowedPattern: '^[\w+=,.@:/-]+$' CallbackSecret: Type: String NoEcho: true Description: One-time secret the stack uses to report back to Netlumi. Set by Netlumi. MinLength: 32 MaxLength: 256 CallbackUrl: Type: String Description: Netlumi endpoint the stack reports to. Set by Netlumi. AllowedPattern: '^https://[^\s]+$' NetlumiAccountId: Type: String Description: The Netlumi AWS account trusted to assume the scanner role. Set by Netlumi. AllowedPattern: '^\d{12}$' AccountType: Type: String Description: >- 'single' connects this account. 'organization' connects the management account and onboards member accounts through a StackSet. Default: single AllowedValues: - single - organization Regions: Type: CommaDelimitedList Description: >- Regions to forward real-time events from (comma separated). IAM and other global events are always forwarded from us-east-1, and console sign-ins also from us-east-2, us-west-2, eu-north-1 and ap-southeast-2 (where AWS records some of them). RealtimeForwarding: Type: String Description: >- 'enabled' creates the EventBridge rules that forward real-time changes, sign-ins and threat signals to Netlumi. 'disabled' creates none (Netlumi then only scans). Set by Netlumi for its environment; an update turns it on later. Default: enabled AllowedValues: - enabled - disabled EnableCloudTrail: Type: String Description: >- Create a multi-region CloudTrail trail if this account has none that is logging. Real-time change detection needs CloudTrail management events. Default: 'false' AllowedValues: - 'true' - 'false' ForwardAIInvocationEvents: Type: String Description: >- Also forward per-request AI runtime calls (Bedrock InvokeModel/Converse, agent and AgentCore invocations, SageMaker endpoints) of roles and people for real-time abuse alerts. Off by default: they can be millions of events a month, billed to your account by EventBridge. Calls signed with long-term access keys (AKIA...) are always forwarded: stolen keys are how model abuse (LLMjacking) starts. Default: 'false' AllowedValues: - 'true' - 'false' # AI agent monitoring. AgentMonitoring: Type: String Description: >- Log and forward what your AI agents' tool roles do (S3, DynamoDB, Lambda, SES, SNS, SQS, Bedrock and AgentCore calls), only for the roles Netlumi finds acting for an agent. Nothing is logged until Netlumi finds an agent. Billed to your account by CloudTrail ($0.10 per 100,000 events) and EventBridge ($1 per million), and capped by AgentMonitoringMonthlyCapUSD. 'disabled' removes the trail and rules. Default: enabled AllowedValues: - enabled - disabled AgentMonitoringMonthlyCapUSD: Type: Number Description: >- Monthly limit in US dollars for AI agent monitoring in this account. When the month's projected cost passes it, the agent trail stops logging until the 1st of next month. Default: 10 MinValue: 1 MaxValue: 1000 EnableDnsQueryLogging: Type: String Description: >- Log DNS queries of the VPCs in DnsQueryLoggingVpcIds to CloudWatch (/netlumi/route53resolver/query-logs, 30 days; about $0.50/GB, billed to you), to show which workloads call AI provider APIs. Default: 'false' AllowedValues: - 'true' - 'false' DnsQueryLoggingVpcIds: Type: CommaDelimitedList Description: >- VPC ids (in any of the selected Regions) to log DNS queries for when EnableDnsQueryLogging is true (comma separated). Default: '' EksClusters: Type: CommaDelimitedList Description: >- Optional. Up to five EKS clusters in this stack's region Netlumi may read with AmazonEKSViewPolicy (never Secrets). Authentication mode API or API_AND_CONFIG_MAP. Default: '' OrgBootstrapAccess: Type: String Description: >- Organizations only. 'enabled' lets Netlumi use a bootstrap role for 7 days after a create or update, to deploy the Netlumi-Org-Bootstrap StackSet. Set 'removed' once member accounts are connected; new accounts are still connected by the StackSet's automatic deployment. Default: enabled AllowedValues: - enabled - removed ExtraStateObjectArns: Type: CommaDelimitedList Description: >- Optional. S3 object ARNs of Terraform state files whose key does not end in .tfstate (and, for buckets with S3 Bucket Keys, the bucket ARN). Netlumi lists the exact values from your code; leave empty otherwise. Default: '' StateBuckets: Type: CommaDelimitedList Description: >- Optional. Names of S3 buckets holding Terraform state that Netlumi lists under Settings > State buckets. Grants listing and reading only these buckets (and decrypting their objects through S3). Default: '' StateLockDiscovery: Type: String Description: >- 'enabled' lets Netlumi read only the LockID attribute of Terraform DynamoDB lock tables (names containing lock, terraform or tfstate) to find where state is kept. 'disabled' removes the grant. Default: enabled AllowedValues: - enabled - disabled PermissionsBoundaryArn: Type: String Description: >- Optional. The ARN of a permissions boundary policy your organization requires on new IAM roles. Applied to the four roles this stack creates; leave empty when none is required. Default: '' AllowedPattern: '^(arn:aws:iam::(\d{12}|aws):policy/[\w+=,.@/-]{1,512})?$' Conditions: HasExtraStateObjects: !Not [!Equals [!Join ['', !Ref ExtraStateObjectArns], '']] HasStateBuckets: !Not [!Equals [!Join ['', !Ref StateBuckets], '']] DiscoverStateLocks: !Equals [!Ref StateLockDiscovery, enabled] IsOrganization: !Equals [!Ref AccountType, organization] CreateOrgBootstrapRole: !And - !Condition IsOrganization - !Equals [!Ref OrgBootstrapAccess, enabled] CreateCloudTrailBucket: !Equals [!Ref EnableCloudTrail, 'true'] CreateDnsQueryLogging: !Equals [!Ref EnableDnsQueryLogging, 'true'] HasEksClusters: !Not [!Equals [!Join ['', !Ref EksClusters], '']] HasPermissionsBoundary: !Not [!Equals [!Ref PermissionsBoundaryArn, '']] AgentMonitoringEnabled: !Equals [!Ref AgentMonitoring, enabled] Resources: # Reports the start, and the first failure on rollback. NetlumiStackWatch: Type: Custom::NetlumiStackWatch # Deleted before the log group its function writes to. DependsOn: NetlumiSetupLogGroup Properties: ServiceToken: !GetAtt NetlumiSetupFunction.Arn # Fail after 15 minutes without an answer, not CloudFormation's default hour. ServiceTimeout: 900 Mode: watch RegistrationId: !Ref RegistrationId CallbackSecret: !Ref CallbackSecret CallbackUrl: !Ref CallbackUrl TemplateVersion: '23' # Explicit Deny on data-plane reads and credential-issuing actions Netlumi # never calls, whichever policy would allow them (a Deny overrides any # Allow). The scanner role has SecurityAudit plus NetlumiScannerReads and # NetlumiScannerReadsScoped. The scanner role does not use ReadOnlyAccess. # The description below still names it: a managed policy's description # cannot change without replacing the policy. NetlumiDataReadGuard: Type: AWS::IAM::ManagedPolicy DependsOn: NetlumiStackWatch Properties: ManagedPolicyName: !Sub 'NetlumiDataReadGuard-${AWS::AccountId}' Description: Netlumi scanner role - deny data reads and credential-issuing actions granted by ReadOnlyAccess or SecurityAudit. PolicyDocument: Version: '2012-10-17' Statement: - Sid: DenyDataReadsGrantedByManagedPolicies Effect: Deny Action: - 'acm:GetAcmeExternalAccountBindingCredentials' - 'appconfig:GetConfiguration' - 'appconfig:GetHostedConfigurationVersion' - 'appsync:GetDataSourceIntrospection' - 'appsync:GetGraphqlApiEnvironmentVariables' - 'appsync:ListApiKeys' - 'athena:GetCalculationExecutionCode' - 'athena:GetQueryResults' - 'athena:GetQueryResultsStream' - 'athena:GetSessionEndpoint' - 'bedrock:GetAgentMemory' - 'bedrock:GetInvocationStep' - 'bedrock-agentcore:GetEvent' - 'bedrock-agentcore:GetMemoryRecord' - 'bedrock-agentcore:ListEvents' - 'bedrock-agentcore:ListMemoryRecords' - 'bedrock-agentcore:RetrieveMemoryRecords' - 'bedrock-mantle:CallWithBearerToken' - 'cassandra:Select' - 'chime:GetChannelMessage' - 'chime:ListChannelMessages' - 'cloudtrail:GetQueryResults' - 'cloudwatch:GetSpaceCredentials' - 'cloudwatch:GetSpaceCredentialsForOrganization' - 'codeartifact:GetAuthorizationToken' - 'codeartifact:GetPackageVersionAsset' - 'codeartifact:GetPackageVersionReadme' - 'codeartifact:ReadFromRepository' - 'codebuild:DescribeTestCases' - 'codecommit:BatchGetCommits' - 'codecommit:DescribeMergeConflicts' - 'codecommit:GetBlob' - 'codecommit:GetCommentsForComparedCommit' - 'codecommit:GetCommentsForPullRequest' - 'codecommit:GetCommit' - 'codecommit:GetDifferences' - 'codecommit:GetFile' - 'codecommit:GetFolder' - 'codecommit:GetMergeConflicts' - 'codecommit:GitPull' - 'cognito-identity:GetCredentialsForIdentity' - 'cognito-identity:GetOpenIdToken' - 'cognito-identity:GetOpenIdTokenForDeveloperIdentity' - 'cognito-idp:AdminGetUser' - 'cognito-idp:DescribeUserPoolClient' - 'cognito-idp:GetTokensFromRefreshToken' - 'cognito-idp:ListUserPoolClientSecrets' - 'cognito-idp:ListUsers' - 'cognito-idp:ListUsersInGroup' - 'connect:GetContactAttributes' - 'connect:GetFederationToken' - 'dax:BatchGetItem' - 'dax:GetItem' - 'dax:Query' - 'dax:Scan' - 'dynamodb:BatchGetItem' - 'dynamodb:GetItem' - 'dynamodb:GetRecords' - 'dynamodb:GetShardIterator' - 'dynamodb:PartiQLSelect' - 'dynamodb:Query' - 'dynamodb:ReadDataForReplication' - 'dynamodb:SearchVectors' - 'ec2:GetConsoleOutput' - 'ec2:GetConsoleScreenshot' - 'ec2:GetLaunchTemplateData' - 'ec2:GetPasswordData' - 'ecr:BatchGetImage' - 'ecr:GetAuthorizationToken' - 'ecr:GetDownloadUrlForLayer' - 'ecr-public:GetAuthorizationToken' - 'elasticbeanstalk:RetrieveEnvironmentInfo' - 'es:ESHttpGet' - 'es:ESHttpHead' - 'forecast:QueryForecast' - 'forecast:QueryWhatIfForecast' - 'gamelift:GetComputeAccess' - 'gamelift:GetComputeAuthToken' - 'gamelift:GetInstanceAccess' - 'glacier:GetJobOutput' - 'healthlake:ReadResource' - 'healthlake:SearchWithGet' - 'healthlake:SearchWithPost' - 'iot:GetRetainedMessage' - 'iot:GetThingShadow' - 'iot:ListRetainedMessages' - 'iotanalytics:GetDatasetContent' - 'iotsitewise:GetAssetPropertyAggregates' - 'iotsitewise:GetAssetPropertyValue' - 'iotsitewise:GetAssetPropertyValueHistory' - 'iotsitewise:GetInterpolatedAssetPropertyValues' - 'kendra:Query' - 'kinesis:GetRecords' - 'kinesis:GetShardIterator' - 'kinesisvideo:GetClip' - 'kinesisvideo:GetDASHStreamingSessionURL' - 'kinesisvideo:GetHLSStreamingSessionURL' - 'kinesisvideo:GetImages' - 'kinesisvideo:GetMedia' - 'kinesisvideo:GetMediaForFragmentList' - 'lex:GetSession' - 'lex:GetUtterancesView' - 'license-manager:GetAccessToken' - 'lightsail:GetRelationalDatabaseLogEvents' - 'logs:FilterLogEvents' - 'logs:GetLogEvents' - 'logs:GetLogFields' - 'logs:GetLogGroupFields' - 'logs:GetLogRecord' - 'logs:StartLiveTail' - 'mediapackagev2:GetObject' - 'mediastore:GetObject' - 'mobiletargeting:GetEndpoint' - 'mobiletargeting:GetUserEndpoints' - 'omics:GetReadSet' - 'personalize:GetPersonalizedRanking' - 'personalize:GetRecommendations' - 'qldb:GetBlock' - 'qldb:GetDigest' - 'qldb:GetRevision' - 'rds:DownloadCompleteDBLogFile' - 'rds:DownloadDBLogFilePortion' - 'rekognition:GetCelebrityRecognition' - 'rekognition:GetContentModeration' - 'rekognition:GetFaceDetection' - 'rekognition:GetFaceSearch' - 'rekognition:GetLabelDetection' - 'rekognition:GetPersonTracking' - 'rekognition:GetSegmentDetection' - 'rekognition:GetTextDetection' - 'rekognition:ListFaces' - 'rekognition:ListUsers' - 'route53globalresolver:ListAccessTokens' - 's3:GetDataAccess' - 's3:GetObjectAnnotation' - 's3:GetObjectTorrent' - 's3:GetObjectVersionAnnotation' - 's3:GetObjectVersionForReplication' - 's3:GetObjectVersionTorrent' - 's3-object-lambda:GetObject' - 's3-object-lambda:GetObjectVersion' - 's3-outposts:GetObject' - 's3-outposts:GetObjectVersion' - 's3-outposts:GetObjectVersionForReplication' - 's3express:CreateSession' - 's3tables:GetTableData' - 'sdb:GetAttributes' - 'sdb:Select' - 'ses:GetContact' - 'ses:GetMessageInsights' - 'ses:ListContacts' - 'sqs:ReceiveMessage' - 'ssm:GetAccessToken' - 'ssm:GetCommandInvocation' - 'ssm:GetParameterHistory' - 'ssm:ListCommandInvocations' - 'sso-directory:ListBearerTokens' - 'states:DescribeExecution' - 'states:GetExecutionHistory' - 'storagegateway:DescribeChapCredentials' - 'sts:GetSessionToken' - 'support:DescribeAttachment' - 'support:DescribeCommunications' - 'swf:GetWorkflowExecutionHistory' - 'transcribe:GetCallAnalyticsJob' - 'transcribe:GetMedicalScribeJob' - 'transcribe:GetMedicalTranscriptionJob' - 'transcribe:GetTranscriptionJob' - 'wafv2:GetDecryptedAPIKey' - 'wafv2:ListAPIKeys' - 'workdocs:GetDocument' - 'workdocs:GetDocumentVersion' - 'xray:BatchGetTraceSummaryById' - 'xray:BatchGetTraces' - 'xray:GetRetrievedTracesGraph' - 'xray:GetTraceSummaries' - 'xray:ListRetrievedTraces' Resource: '*' # Content, personal data and download URLs that SecurityAudit # grants and Netlumi never reads. - Sid: DenyContentAndPersonalDataReads Effect: Deny Action: - 'bedrock:GetKnowledgeBaseDocuments' - 'chime:ListAppInstanceUsers' - 'chime:ListMeetingEvents' - 'chime:ListProxySessions' - 'chime:ListUsers' - 'chime:ListVoiceProfiles' - 'cleanrooms:GetProtectedQuery' - 'codestar:DescribeUserProfile' - 'codestar:ListUserProfiles' - 'cognito-sync:ListRecords' - 'identitystore:GetUserId' - 'identitystore:ListUsers' - 'lambda:GetFunction' - 'lambda:GetLayerVersion' - 'quicksight:DescribeUser' - 'quicksight:ListUsers' - 'security-ir:GetCaseAttachmentDownloadUrl' Resource: '*' # Object reads only for Terraform state and opt-in DNS query logs, KMS # decrypt only through S3 and Lambda, lock tables' LockID only, and more # data reads no managed policy grants today. NetlumiDataReadGuardScoped: Type: AWS::IAM::ManagedPolicy DependsOn: NetlumiStackWatch Properties: ManagedPolicyName: !Sub 'NetlumiDataReadGuardScoped-${AWS::AccountId}' Description: Netlumi scanner role - object, KMS and lock-table reads only where Netlumi reads Terraform state and opt-in DNS logs. PolicyDocument: Version: '2012-10-17' Statement: - Sid: DenyObjectReadsOutsideState Effect: Deny Action: - 's3:GetObject' - 's3:GetObjectVersion' # Empty-string branches, not duplicate entries, when a list is empty. NotResource: !Split - ',' - !Join - '' - - !Sub 'arn:${AWS::Partition}:s3:::*/*.tfstate,arn:${AWS::Partition}:s3:::*tfstate*/*,arn:${AWS::Partition}:s3:::*terraform-state*/*,arn:${AWS::Partition}:s3:::*tf-state*/*,arn:${AWS::Partition}:s3:::*terraform.state*/*,arn:${AWS::Partition}:s3:::*/AWSLogs/${AWS::AccountId}/vpcdnsquerylogs/*' - !If [HasStateBuckets, !Join ['', [',arn:aws:s3:::', !Join ['/*,arn:aws:s3:::', !Ref StateBuckets], '/*']], ''] - !If [HasExtraStateObjects, !Join ['', [',', !Join [',', !Ref ExtraStateObjectArns]]], ''] - Sid: DenyDecryptExceptS3AndLambda Effect: Deny Action: 'kms:Decrypt' Resource: '*' Condition: StringNotLike: 'kms:ViaService': - 's3.*.amazonaws.com' - 'lambda.*.amazonaws.com' - !If - DiscoverStateLocks - Sid: DenyItemScanOutsideLockTables Effect: Deny Action: 'dynamodb:Scan' NotResource: - !Sub 'arn:${AWS::Partition}:dynamodb:*:${AWS::AccountId}:table/*lock*' - !Sub 'arn:${AWS::Partition}:dynamodb:*:${AWS::AccountId}:table/*terraform*' - !Sub 'arn:${AWS::Partition}:dynamodb:*:${AWS::AccountId}:table/*tfstate*' - Sid: DenyItemScanOutsideLockTables Effect: Deny Action: 'dynamodb:Scan' Resource: '*' - Sid: DenyScanOfOtherAttributes Effect: Deny Action: 'dynamodb:Scan' Resource: '*' Condition: 'ForAnyValue:StringNotEquals': 'dynamodb:Attributes': - LockID - Sid: DenyScanWithoutProjection Effect: Deny Action: 'dynamodb:Scan' Resource: '*' Condition: 'Null': 'dynamodb:Attributes': 'true' - Sid: DenyDataReadsNotGrantedToday Effect: Deny Action: - 'aoss:APIAccessAll' - 'appconfig:GetLatestConfiguration' - 'appsync:GraphQL' - 'athena:ExportNotebook' - 'bedrock:GetSession' - 'bedrock:ListInvocationSteps' - 'bedrock:Retrieve' - 'bedrock-agentcore:GetResourceApiKey' - 'bedrock-agentcore:GetResourceOauth2Token' - 'bedrock-agentcore:GetWorkloadAccessToken' - 'datazone:GetEnvironmentCredentials' - 'ebs:GetSnapshotBlock' - 'ebs:ListChangedBlocks' - 'ebs:ListSnapshotBlocks' - 'elasticmapreduce:GetClusterSessionCredentials' - 'emr-containers:GetManagedEndpointSessionCredentials' - 'geo:GetDevicePosition' - 'geo:GetDevicePositionHistory' - 'geo:ListDevicePositions' - 'iotsitewise:BatchGetAssetPropertyAggregates' - 'iotsitewise:BatchGetAssetPropertyValue' - 'iotsitewise:BatchGetAssetPropertyValueHistory' - 'iotsitewise:ExecuteQuery' - 'kafka-cluster:ReadData' - 'kendra:Retrieve' - 'kinesis:SubscribeToShard' - 'lakeformation:GetDataAccess' - 'lakeformation:GetTemporaryGlueTableCredentials' - 'lightsail:DownloadDefaultKeyPair' - 'lightsail:GetContainerLog' - 'lightsail:GetInstanceAccessDetails' - 'lightsail:GetRelationalDatabaseMasterUserPassword' - 'logs:Unmask' - 'macie2:GetSensitiveDataOccurrences' - 'neptune-db:ReadDataViaQuery' - 'qbusiness:ListMessages' - 'rds-data:BatchExecuteStatement' - 'rds-data:ExecuteStatement' - 'redshift:GetClusterCredentials' - 'redshift:GetClusterCredentialsWithIAM' - 'redshift-data:GetStatementResult' - 'redshift-serverless:GetCredentials' - 'sagemaker:BatchGetRecord' - 'sagemaker:GetRecord' - 'secretsmanager:BatchGetSecretValue' - 'secretsmanager:GetSecretValue' - 'textract:GetDocumentAnalysis' - 'textract:GetDocumentTextDetection' - 'textract:GetExpenseAnalysis' - 'textract:GetLendingAnalysis' - 'timestream:Select' - 'workdocs:DownloadDocumentVersion' - 'workmailmessageflow:GetRawMessageContent' Resource: '*' # Netlumi assumes this role from its own account to scan (read-only). # The reads Netlumi makes that SecurityAudit does not grant, exactly. # The scanner role does not use ReadOnlyAccess. Netlumi checks that the # list equals the calls its scanner makes. NetlumiScannerReads: Type: AWS::IAM::ManagedPolicy DependsOn: NetlumiStackWatch Properties: ManagedPolicyName: !Sub 'NetlumiScannerReads-${AWS::AccountId}' Description: Netlumi scanner role - the configuration reads Netlumi makes that SecurityAudit does not grant. PolicyDocument: Version: '2012-10-17' Statement: - Sid: ReadsNotInSecurityAudit Effect: Allow Action: - 'aoss:BatchGetCollection' - 'aoss:GetAccessPolicy' - 'aoss:GetSecurityPolicy' - 'aoss:ListAccessPolicies' - 'aoss:ListCollections' - 'aoss:ListSecurityPolicies' - 'athena:GetDataCatalog' - 'backup:DescribeProtectedResource' - 'backup:ListRecoveryPointsByBackupVault' - 'batch:DescribeJobQueues' - 'bedrock:GetPrompt' - 'cloudhsm:DescribeClusters' - 'cloudwatch:GetMetricStatistics' - 'codebuild:BatchGetBuilds' - 'codebuild:BatchGetReportGroups' - 'codebuild:ListBuildsForProject' - 'codebuild:ListReportGroups' - 'codepipeline:ListTagsForResource' - 'ds:ListCertificates' - 'ds:ListLogSubscriptions' - 'eks:DescribeAddon' - 'glue:GetMLTransforms' - 'iot:GetTopicRule' - 'kendra:DescribeDataSource' - 'kinesis:GetResourcePolicy' - 'macie2:GetAutomatedDiscoveryConfiguration' - 'macie2:GetFindings' - 'macie2:GetMacieSession' - 'mediaconvert:ListJobTemplates' - 'mediaconvert:ListPresets' - 'mediaconvert:ListQueues' - 'medialive:DescribeChannel' - 'medialive:DescribeMultiplex' - 'medialive:ListInputs' - 'medialive:ListMultiplexes' - 'redshift-serverless:ListNamespaces' - 'resource-explorer-2:ListIndexes' - 'securitylake:ListDataLakes' - 'servicecatalog:DescribePortfolioShares' - 'servicecatalog:ListOrganizationPortfolioAccess' - 'servicecatalog:ListPortfolios' - 'ses:ListEmailIdentities' - 'sqs:GetQueueUrl' - 'ssm:GetDocument' - 'ssm:GetMaintenanceWindow' - 'ssm:GetPatchBaseline' - 'states:DescribeActivity' - 'states:ListActivities' - 'wafv2:GetIPSet' - 'wafv2:GetRuleGroup' Resource: '*' # GetQueryResults and StopQuery have no resource-level permissions: they # take a query id, and StartQuery (NetlumiScannerReadsScoped) decides # which log groups a query can read. - Sid: LogsInsightsQueryResults Effect: Allow Action: - 'logs:GetQueryResults' - 'logs:StopQuery' Resource: '*' # Reads limited to this account's resources, the API Gateway paths # SecurityAudit leaves out, and the opt-in Resolver query logs. NetlumiScannerReadsScoped: Type: AWS::IAM::ManagedPolicy DependsOn: NetlumiStackWatch Properties: ManagedPolicyName: !Sub 'NetlumiScannerReadsScoped-${AWS::AccountId}' Description: Netlumi scanner role - reads of this account's log groups, parameters, Resource Explorer views, API Gateway paths and opt-in Resolver query logs. PolicyDocument: Version: '2012-10-17' Statement: # AI egress: aggregate Logs Insights queries on Route 53 Resolver # query-log groups only, in any region of this account: the group # EnableDnsQueryLogging creates (/netlumi/route53resolver/query-logs) # and groups you name under /aws/route53resolver/. - Sid: LogsInsightsQueries Effect: Allow Action: 'logs:StartQuery' Resource: - !Sub 'arn:${AWS::Partition}:logs:*:${AWS::AccountId}:log-group:/netlumi/route53resolver/*' - !Sub 'arn:${AWS::Partition}:logs:*:${AWS::AccountId}:log-group:/aws/route53resolver/*' # String values for the plaintext-secret rules; SecureStrings stay undecryptable (KMS Deny). - Sid: PlaintextParameters Effect: Allow Action: 'ssm:GetParameters' Resource: !Sub 'arn:${AWS::Partition}:ssm:*:${AWS::AccountId}:parameter/*' # Unscanned-region counts, only where the account has an index (default view). - Sid: ResourceExplorerSearch Effect: Allow Action: 'resource-explorer-2:Search' Resource: !Sub 'arn:${AWS::Partition}:resource-explorer-2:*:${AWS::AccountId}:view/*' # GetRestApi, GetApiKey(s), GetUsagePlan(s): paths SecurityAudit leaves out. - Sid: ApiGatewayReadsNotInSecurityAudit Effect: Allow Action: 'apigateway:GET' Resource: - !Sub 'arn:${AWS::Partition}:apigateway:*::/restapis/*' - !Sub 'arn:${AWS::Partition}:apigateway:*::/apikeys' - !Sub 'arn:${AWS::Partition}:apigateway:*::/apikeys/*' - !Sub 'arn:${AWS::Partition}:apigateway:*::/usageplans' - !Sub 'arn:${AWS::Partition}:apigateway:*::/usageplans/*' # This account's opt-in Route 53 Resolver query logs (AI egress). - Sid: ResolverQueryLogObjects Effect: Allow Action: 's3:GetObject' Resource: !Sub 'arn:${AWS::Partition}:s3:::*/AWSLogs/${AWS::AccountId}/vpcdnsquerylogs/*' NetlumiScannerRole: Type: AWS::IAM::Role DependsOn: NetlumiStackWatch Properties: RoleName: !Sub 'NetlumiScannerRole-${AWS::AccountId}' PermissionsBoundary: !If [HasPermissionsBoundary, !Ref PermissionsBoundaryArn, !Ref 'AWS::NoValue'] AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Principal: AWS: !Sub 'arn:${AWS::Partition}:iam::${NetlumiAccountId}:root' Action: 'sts:AssumeRole' Condition: StringEquals: 'sts:ExternalId': !Ref ExternalId # SecurityAudit plus Netlumi's exact grants. # The scanner role does not use ReadOnlyAccess. ManagedPolicyArns: - !Sub 'arn:${AWS::Partition}:iam::aws:policy/SecurityAudit' - !Ref NetlumiScannerReads - !Ref NetlumiScannerReadsScoped - !Ref NetlumiDataReadGuard - !Ref NetlumiDataReadGuardScoped Policies: - PolicyName: NetlumiOrganizationsReadAccess PolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Action: - 'organizations:DescribeOrganization' - 'organizations:DescribeOrganizationalUnit' - 'organizations:DescribePolicy' - 'organizations:ListAccounts' - 'organizations:ListAWSServiceAccessForOrganization' - 'organizations:ListChildren' - 'organizations:ListDelegatedAdministrators' - 'organizations:ListDelegatedServicesForAccount' - 'organizations:ListOrganizationalUnitsForParent' - 'organizations:ListParents' - 'organizations:ListPolicies' - 'organizations:ListPoliciesForTarget' - 'organizations:ListRoots' - 'organizations:ListTargetsForPolicy' Resource: '*' # Never bedrock-agentcore:Get*: it includes reads of stored API keys and tokens. - PolicyName: NetlumiAIServicesReadAccess PolicyDocument: Version: '2012-10-17' Statement: - Sid: BedrockAgentCoreConfigurationRead Effect: Allow Action: - 'bedrock-agentcore:ListGateways' - 'bedrock-agentcore:GetGateway' - 'bedrock-agentcore:ListGatewayTargets' - 'bedrock-agentcore:GetGatewayTarget' - 'bedrock-agentcore:ListAgentRuntimes' - 'bedrock-agentcore:GetAgentRuntime' - 'bedrock-agentcore:ListCodeInterpreters' - 'bedrock-agentcore:GetCodeInterpreter' - 'bedrock-agentcore:ListBrowsers' - 'bedrock-agentcore:GetBrowser' - 'bedrock-agentcore:ListMemories' - 'bedrock-agentcore:GetMemory' - 'bedrock-agentcore:GetTokenVault' Resource: '*' # Inspector finding details and image-to-cluster mapping (not in SecurityAudit). - PolicyName: NetlumiInspectorReadAccess PolicyDocument: Version: '2012-10-17' Statement: - Sid: InspectorFindingContext Effect: Allow Action: - 'inspector2:BatchGetFindingDetails' - 'inspector2:GetClustersForImage' Resource: '*' - PolicyName: NetlumiCodeToCloudAccess PolicyDocument: Version: '2012-10-17' Statement: # Terraform state objects only, by key. - Sid: TerraformStateAccess Effect: Allow Action: - 's3:GetObject' - 's3:GetObjectVersion' Resource: - !Sub 'arn:${AWS::Partition}:s3:::*/*.tfstate' - !Sub 'arn:${AWS::Partition}:s3:::*tfstate*/*' - !Sub 'arn:${AWS::Partition}:s3:::*terraform-state*/*' - !Sub 'arn:${AWS::Partition}:s3:::*tf-state*/*' - !Sub 'arn:${AWS::Partition}:s3:::*terraform.state*/*' - Sid: TerraformStateBucketList Effect: Allow Action: - 's3:ListBucket' Resource: - !Sub 'arn:${AWS::Partition}:s3:::*tfstate*' - !Sub 'arn:${AWS::Partition}:s3:::*terraform-state*' - !Sub 'arn:${AWS::Partition}:s3:::*tf-state*' - !Sub 'arn:${AWS::Partition}:s3:::*terraform.state*' # SSE-KMS state, only through S3. - Sid: TerraformStateDecrypt Effect: Allow Action: - 'kms:Decrypt' Resource: '*' Condition: StringLike: 'kms:ViaService': 's3.*.amazonaws.com' 'kms:EncryptionContext:aws:s3:arn': - !Sub 'arn:${AWS::Partition}:s3:::*/*.tfstate' - !Sub 'arn:${AWS::Partition}:s3:::*tfstate*' - !Sub 'arn:${AWS::Partition}:s3:::*terraform-state*' - !Sub 'arn:${AWS::Partition}:s3:::*tf-state*' - !Sub 'arn:${AWS::Partition}:s3:::*terraform.state*' - !If - HasExtraStateObjects - Sid: TerraformStateExtraObjects Effect: Allow Action: - 's3:GetObject' - 's3:GetObjectVersion' Resource: !Ref ExtraStateObjectArns - !Ref AWS::NoValue - !If - HasExtraStateObjects - Sid: TerraformStateExtraDecrypt Effect: Allow Action: - 'kms:Decrypt' Resource: '*' Condition: StringLike: 'kms:ViaService': 's3.*.amazonaws.com' 'kms:EncryptionContext:aws:s3:arn': !Ref ExtraStateObjectArns - !Ref AWS::NoValue # StateBuckets: only the named buckets. Netlumi assumes this role # from the aws partition, so the ARNs are aws-partition ARNs # (Fn::Join delimiters must be literal strings). - !If - HasStateBuckets - Sid: TerraformStateBucketsList Effect: Allow Action: - 's3:ListBucket' Resource: !Split [',', !Join ['', ['arn:aws:s3:::', !Join [',arn:aws:s3:::', !Ref StateBuckets]]]] - !Ref AWS::NoValue - !If - HasStateBuckets - Sid: TerraformStateBucketsRead Effect: Allow Action: - 's3:GetObject' - 's3:GetObjectVersion' Resource: !Split [',', !Join ['', ['arn:aws:s3:::', !Join ['/*,arn:aws:s3:::', !Ref StateBuckets], '/*']]] - !Ref AWS::NoValue # Object ARNs, and bucket ARNs for S3 Bucket Keys. - !If - HasStateBuckets - Sid: TerraformStateBucketsDecrypt Effect: Allow Action: - 'kms:Decrypt' Resource: '*' Condition: StringLike: 'kms:ViaService': 's3.*.amazonaws.com' 'kms:EncryptionContext:aws:s3:arn': !Split [',', !Join ['', ['arn:aws:s3:::', !Join [',arn:aws:s3:::', !Ref StateBuckets], ',arn:aws:s3:::', !Join ['/*,arn:aws:s3:::', !Ref StateBuckets], '/*']]] - !Ref AWS::NoValue # Terraform lock tables: LockID ("bucket/key") names state # locations. StateLockDiscovery=disabled removes it. - !If - DiscoverStateLocks - Sid: TerraformStateLockTableRead Effect: Allow Action: - 'dynamodb:Scan' - 'dynamodb:DescribeTable' Resource: - !Sub 'arn:${AWS::Partition}:dynamodb:*:${AWS::AccountId}:table/*lock*' - !Sub 'arn:${AWS::Partition}:dynamodb:*:${AWS::AccountId}:table/*terraform*' - !Sub 'arn:${AWS::Partition}:dynamodb:*:${AWS::AccountId}:table/*tfstate*' - !Ref AWS::NoValue - Sid: AWSConfigAccess Effect: Allow Action: - 'config:DescribeConfigurationRecorders' - 'config:DescribeDeliveryChannels' - 'config:GetResourceConfigHistory' - 'config:ListDiscoveredResources' - 'config:SelectResourceConfig' - 'config:DescribeConfigurationRecorderStatus' Resource: '*' - Sid: CodePipelineAccess Effect: Allow Action: - 'codepipeline:ListPipelines' - 'codebuild:ListProjects' - 'codedeploy:ListApplications' - 'codedeploy:ListDeployments' Resource: '*' # Organizations only: deploys the Netlumi-Org-Bootstrap StackSet, until # BootstrapNotAfter (7 days after the last create/update). NetlumiOrgBootstrapRole: Type: AWS::IAM::Role Condition: CreateOrgBootstrapRole DependsOn: NetlumiStackWatch Properties: RoleName: !Sub 'NetlumiOrgBootstrapRole-${AWS::AccountId}' PermissionsBoundary: !If [HasPermissionsBoundary, !Ref PermissionsBoundaryArn, !Ref 'AWS::NoValue'] MaxSessionDuration: 3600 AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Principal: AWS: !Sub 'arn:${AWS::Partition}:iam::${NetlumiAccountId}:root' Action: 'sts:AssumeRole' Condition: StringEquals: 'sts:ExternalId': !Ref ExternalId DateLessThan: 'aws:CurrentTime': !GetAtt NetlumiStackWatch.BootstrapNotAfter Policies: - PolicyName: NetlumiOrgBootstrapPermissions PolicyDocument: Version: '2012-10-17' Statement: - Sid: ReadOrganization Effect: Allow Action: - 'organizations:DescribeOrganization' - 'organizations:DescribePolicy' - 'organizations:ListAccounts' - 'organizations:ListParents' - 'organizations:ListRoots' - 'organizations:ListOrganizationalUnitsForParent' - 'organizations:ListPoliciesForTarget' - 'organizations:ListDelegatedAdministrators' - 'organizations:ListAWSServiceAccessForOrganization' - 'lambda:GetAccountSettings' - 'cloudformation:ListStackSets' Resource: '*' - Sid: NetlumiStackSetOnly Effect: Allow Action: - 'cloudformation:CreateStackSet' - 'cloudformation:UpdateStackSet' - 'cloudformation:DescribeStackSet' - 'cloudformation:DescribeStackSetOperation' - 'cloudformation:ListStackSetOperations' - 'cloudformation:StopStackSetOperation' - 'cloudformation:CreateStackInstances' - 'cloudformation:UpdateStackInstances' - 'cloudformation:DescribeStackInstance' - 'cloudformation:ListStackInstances' Resource: - !Sub 'arn:${AWS::Partition}:cloudformation:*:${AWS::AccountId}:stackset/Netlumi-Org-Bootstrap*' - !Sub 'arn:${AWS::Partition}:cloudformation:*:${AWS::AccountId}:stackset-target/Netlumi-Org-Bootstrap*' - !Sub 'arn:${AWS::Partition}:cloudformation:*:*:type/resource/*' # Used by the forwarding rules (each selected region, and us-east-1). NetlumiEventBridgeForwarderRole: Type: AWS::IAM::Role DependsOn: NetlumiStackWatch Properties: RoleName: !Sub 'NetlumiEventBridgeForwarderRole-${AWS::AccountId}' PermissionsBoundary: !If [HasPermissionsBoundary, !Ref PermissionsBoundaryArn, !Ref 'AWS::NoValue'] AssumeRolePolicyDocument: Version: '2012-10-17' Statement: # Only this account's Netlumi rules (confused deputy). - Effect: Allow Principal: Service: events.amazonaws.com Action: 'sts:AssumeRole' Condition: StringEquals: 'aws:SourceAccount': !Ref 'AWS::AccountId' ArnLike: # Every forwarding rule: writes, -writes-N, -reads, -failures, -other and the others. 'aws:SourceArn': - !Sub 'arn:${AWS::Partition}:events:*:${AWS::AccountId}:rule/netlumi-cloudtrail-forwarder-${AWS::AccountId}' - !Sub 'arn:${AWS::Partition}:events:*:${AWS::AccountId}:rule/netlumi-cloudtrail-forwarder-${AWS::AccountId}-*' Policies: - PolicyName: NetlumiEventBridgeForwarderPolicy PolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Action: 'events:PutEvents' Resource: - !Sub 'arn:${AWS::Partition}:events:*:${NetlumiAccountId}:event-bus/netlumi-realtime-ingress' - !Sub 'arn:${AWS::Partition}:events:us-east-1:${NetlumiAccountId}:event-bus/netlumi-realtime-events' # Kept on stack deletion: it holds your audit logs. NetlumiCloudTrailBucket: Type: AWS::S3::Bucket Condition: CreateCloudTrailBucket # After the watch: a refused CreateBucket (an SCP) is # reported to Netlumi with the stack's first failure, never silent. DependsOn: NetlumiStackWatch DeletionPolicy: Retain UpdateReplacePolicy: Retain Properties: BucketEncryption: ServerSideEncryptionConfiguration: - ServerSideEncryptionByDefault: SSEAlgorithm: AES256 PublicAccessBlockConfiguration: BlockPublicAcls: true BlockPublicPolicy: true IgnorePublicAcls: true RestrictPublicBuckets: true VersioningConfiguration: Status: Enabled # Netlumi-managed hardening: # ACLs off (CloudTrail's bucket-owner-full-control writes still succeed); # unfinished uploads and replaced log versions do not accumulate. Current # log objects are kept: how long audit logs are retained is your decision. OwnershipControls: Rules: - ObjectOwnership: BucketOwnerEnforced LifecycleConfiguration: Rules: - Id: NetlumiTrailHousekeeping Status: Enabled AbortIncompleteMultipartUpload: DaysAfterInitiation: 7 NoncurrentVersionExpiration: NoncurrentDays: 90 NetlumiCloudTrailBucketPolicy: Type: AWS::S3::BucketPolicy Condition: CreateCloudTrailBucket DeletionPolicy: Retain UpdateReplacePolicy: Retain Properties: Bucket: !Ref NetlumiCloudTrailBucket PolicyDocument: Version: '2012-10-17' Statement: - Sid: AWSCloudTrailAclCheck Effect: Allow Principal: Service: cloudtrail.amazonaws.com Action: 's3:GetBucketAcl' Resource: !GetAtt NetlumiCloudTrailBucket.Arn Condition: StringEquals: 'aws:SourceArn': !Sub 'arn:${AWS::Partition}:cloudtrail:${AWS::Region}:${AWS::AccountId}:trail/netlumi-trail' - Sid: AWSCloudTrailWrite Effect: Allow Principal: Service: cloudtrail.amazonaws.com Action: 's3:PutObject' Resource: !Sub '${NetlumiCloudTrailBucket.Arn}/AWSLogs/${AWS::AccountId}/*' Condition: StringEquals: 's3:x-amz-acl': bucket-owner-full-control 'aws:SourceArn': !Sub 'arn:${AWS::Partition}:cloudtrail:${AWS::Region}:${AWS::AccountId}:trail/netlumi-trail' - Sid: DenyInsecureTransport Effect: Deny Principal: '*' Action: 's3:*' Resource: - !GetAtt NetlumiCloudTrailBucket.Arn - !Sub '${NetlumiCloudTrailBucket.Arn}/*' Condition: Bool: 'aws:SecureTransport': 'false' # The log bucket of netlumi-agent-activity, the data-events-only # trail the setup function creates once Netlumi finds an AI agent (an empty # bucket costs nothing). Kept on stack deletion: it holds your audit logs, # which expire after 30 days. NetlumiAgentActivityBucket: Type: AWS::S3::Bucket Condition: AgentMonitoringEnabled DependsOn: NetlumiStackWatch DeletionPolicy: Retain UpdateReplacePolicy: Retain Properties: BucketEncryption: ServerSideEncryptionConfiguration: - ServerSideEncryptionByDefault: SSEAlgorithm: AES256 PublicAccessBlockConfiguration: BlockPublicAcls: true BlockPublicPolicy: true IgnorePublicAcls: true RestrictPublicBuckets: true VersioningConfiguration: Status: Enabled OwnershipControls: Rules: - ObjectOwnership: BucketOwnerEnforced LifecycleConfiguration: Rules: - Id: NetlumiAgentActivityExpiry Status: Enabled ExpirationInDays: 30 NoncurrentVersionExpiration: NoncurrentDays: 7 AbortIncompleteMultipartUpload: DaysAfterInitiation: 7 NetlumiAgentActivityBucketPolicy: Type: AWS::S3::BucketPolicy Condition: AgentMonitoringEnabled DeletionPolicy: Retain UpdateReplacePolicy: Retain Properties: Bucket: !Ref NetlumiAgentActivityBucket PolicyDocument: Version: '2012-10-17' Statement: - Sid: AWSCloudTrailAclCheck Effect: Allow Principal: Service: cloudtrail.amazonaws.com Action: 's3:GetBucketAcl' Resource: !GetAtt NetlumiAgentActivityBucket.Arn Condition: StringEquals: 'aws:SourceArn': !Sub 'arn:${AWS::Partition}:cloudtrail:${AWS::Region}:${AWS::AccountId}:trail/netlumi-agent-activity' - Sid: AWSCloudTrailWrite Effect: Allow Principal: Service: cloudtrail.amazonaws.com Action: 's3:PutObject' Resource: !Sub '${NetlumiAgentActivityBucket.Arn}/AWSLogs/${AWS::AccountId}/*' Condition: StringEquals: 's3:x-amz-acl': bucket-owner-full-control 'aws:SourceArn': !Sub 'arn:${AWS::Partition}:cloudtrail:${AWS::Region}:${AWS::AccountId}:trail/netlumi-agent-activity' - Sid: DenyInsecureTransport Effect: Deny Principal: '*' Action: 's3:*' Resource: - !GetAtt NetlumiAgentActivityBucket.Arn - !Sub '${NetlumiAgentActivityBucket.Arn}/*' Condition: Bool: 'aws:SecureTransport': 'false' # Named, so the forwarding rules exclude exactly this role. NetlumiSetupLambdaRole: Type: AWS::IAM::Role Properties: # Netlumi-managed hardening: # a name carrying the account id, so Netlumi recognises its own role, and # the exact name real-time rules leave out. RoleName: !Sub 'NetlumiSetupLambdaRole-${AWS::AccountId}' PermissionsBoundary: !If [HasPermissionsBoundary, !Ref PermissionsBoundaryArn, !Ref 'AWS::NoValue'] AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Principal: Service: lambda.amazonaws.com Action: 'sts:AssumeRole' Policies: - PolicyName: NetlumiSetupLambdaPolicy PolicyDocument: Version: '2012-10-17' Statement: # The function writes only to its own log group and cannot # create a log group. With a create grant, logs written while a # failed stack rolls back would re-create the group after # CloudFormation had deleted it, and the next stack in this # account and region would fail with "already exists". - Sid: OwnLogGroup Effect: Allow Action: - 'logs:CreateLogStream' - 'logs:PutLogEvents' Resource: !GetAtt NetlumiSetupLogGroup.Arn - Sid: ForwardingRules Effect: Allow Action: - 'events:PutRule' - 'events:PutTargets' - 'events:DeleteRule' - 'events:RemoveTargets' - 'events:DescribeRule' Resource: !Sub 'arn:${AWS::Partition}:events:*:${AWS::AccountId}:rule/netlumi-*' - Sid: DeliveryProbe Effect: Allow Action: 'events:PutEvents' Resource: - !Sub 'arn:${AWS::Partition}:events:*:${AWS::AccountId}:event-bus/default' - !Sub 'arn:${AWS::Partition}:events:*:${NetlumiAccountId}:event-bus/netlumi-realtime-ingress' - !Sub 'arn:${AWS::Partition}:events:us-east-1:${NetlumiAccountId}:event-bus/netlumi-realtime-events' - Sid: PassForwarderRole Effect: Allow Action: 'iam:PassRole' Resource: !Sub 'arn:${AWS::Partition}:iam::${AWS::AccountId}:role/NetlumiEventBridgeForwarderRole-${AWS::AccountId}' - Sid: OwnStackStatus Effect: Allow Action: - 'cloudformation:DescribeStacks' - 'cloudformation:DescribeStackEvents' Resource: !Ref 'AWS::StackId' - Sid: CloudTrailRead Effect: Allow Action: - 'cloudtrail:DescribeTrails' - 'cloudtrail:GetTrailStatus' - 'cloudtrail:GetEventSelectors' Resource: '*' # Which regions the account has enabled (no # EventBridge call in a region that is not), and whether AWS # Control Tower manages the account (that one role, read only). - Sid: EnabledRegions Effect: Allow Action: 'ec2:DescribeRegions' Resource: '*' - Sid: ControlTowerPresence Effect: Allow Action: 'iam:GetRole' Resource: !Sub 'arn:${AWS::Partition}:iam::${AWS::AccountId}:role/AWSControlTowerExecution' - Sid: NetlumiTrail Effect: Allow Action: - 'cloudtrail:CreateTrail' - 'cloudtrail:StartLogging' - 'cloudtrail:StopLogging' - 'cloudtrail:DeleteTrail' Resource: !Sub 'arn:${AWS::Partition}:cloudtrail:*:${AWS::AccountId}:trail/netlumi-trail' - Sid: OrganizationId Effect: Allow Action: 'organizations:DescribeOrganization' Resource: '*' # Resolver and log-delivery actions have no resource-level permissions. # The log group (your DNS history) is kept on stack deletion. NetlumiDnsQueryLoggingPolicy: Type: AWS::IAM::Policy Condition: CreateDnsQueryLogging Properties: PolicyName: NetlumiDnsQueryLoggingSetup Roles: - !Ref NetlumiSetupLambdaRole PolicyDocument: Version: '2012-10-17' Statement: - Sid: ResolverQueryLogging Effect: Allow Action: - 'route53resolver:CreateResolverQueryLogConfig' - 'route53resolver:DeleteResolverQueryLogConfig' - 'route53resolver:AssociateResolverQueryLogConfig' - 'route53resolver:DisassociateResolverQueryLogConfig' - 'route53resolver:GetResolverQueryLogConfig' - 'route53resolver:ListResolverQueryLogConfigs' - 'route53resolver:ListResolverQueryLogConfigAssociations' - 'ec2:DescribeVpcs' Resource: '*' - Sid: QueryLogGroup Effect: Allow Action: - 'logs:CreateLogGroup' - 'logs:PutRetentionPolicy' Resource: !Sub 'arn:${AWS::Partition}:logs:*:${AWS::AccountId}:log-group:/netlumi/route53resolver/*' - Sid: QueryLogDelivery Effect: Allow Action: - 'logs:CreateLogDelivery' - 'logs:GetLogDelivery' - 'logs:UpdateLogDelivery' - 'logs:DeleteLogDelivery' - 'logs:ListLogDeliveries' - 'logs:PutResourcePolicy' - 'logs:DescribeResourcePolicies' - 'logs:DescribeLogGroups' Resource: '*' # EKS workload read: the setup function gives the scanner # role AmazonEKSViewPolicy (never Secrets) on the listed clusters in this # stack's region, and nothing else. A cluster it cannot reach is a warning, # not a failed stack. NetlumiEksAccessPolicy: Type: AWS::IAM::Policy Condition: HasEksClusters Properties: PolicyName: NetlumiEksAccessSetup Roles: - !Ref NetlumiSetupLambdaRole PolicyDocument: Version: '2012-10-17' Statement: - Sid: ScannerRoleAccessEntryOnly Effect: Allow Action: 'eks:CreateAccessEntry' Resource: !Sub 'arn:${AWS::Partition}:eks:${AWS::Region}:${AWS::AccountId}:cluster/*' Condition: StringEquals: 'eks:principalArn': !Sub 'arn:${AWS::Partition}:iam::${AWS::AccountId}:role/NetlumiScannerRole-${AWS::AccountId}' - Sid: ViewPolicyOnly Effect: Allow Action: 'eks:AssociateAccessPolicy' Resource: !Sub 'arn:${AWS::Partition}:eks:${AWS::Region}:${AWS::AccountId}:access-entry/*/role/${AWS::AccountId}/NetlumiScannerRole-${AWS::AccountId}/*' Condition: StringEquals: 'eks:policyArn': !Sub 'arn:${AWS::Partition}:eks::aws:cluster-access-policy/AmazonEKSViewPolicy' - Sid: ScannerRoleAccessEntryCleanup Effect: Allow Action: - 'eks:DescribeAccessEntry' - 'eks:DeleteAccessEntry' Resource: !Sub 'arn:${AWS::Partition}:eks:${AWS::Region}:${AWS::AccountId}:access-entry/*/role/${AWS::AccountId}/NetlumiScannerRole-${AWS::AccountId}/*' # What the agent sync may do, each scoped to Netlumi's names. # The forwarding-rule actions are already granted on rule/netlumi-*. NetlumiAgentMonitoringPolicy: Type: AWS::IAM::Policy Condition: AgentMonitoringEnabled Properties: PolicyName: NetlumiAgentMonitoring Roles: - !Ref NetlumiSetupLambdaRole PolicyDocument: Version: '2012-10-17' Statement: - Sid: AgentActivityTrailOnly Effect: Allow Action: - 'cloudtrail:CreateTrail' - 'cloudtrail:PutEventSelectors' - 'cloudtrail:StartLogging' - 'cloudtrail:StopLogging' - 'cloudtrail:DeleteTrail' - 'cloudtrail:AddTags' Resource: !Sub 'arn:${AWS::Partition}:cloudtrail:*:${AWS::AccountId}:trail/netlumi-agent-activity' # A listed role must exist in this account (metadata only). - Sid: ListedRolesExist Effect: Allow Action: 'iam:GetRole' Resource: !Sub 'arn:${AWS::Partition}:iam::${AWS::AccountId}:role/*' # The cost cap reads the agent rules' MatchedEvents (the action takes no resource). - Sid: AgentRuleVolume Effect: Allow Action: 'cloudwatch:GetMetricStatistics' Resource: '*' # Only this account's list, in Netlumi's bucket. - Sid: NetlumiAgentList Effect: Allow Action: 's3:GetObject' Resource: !Sub 'arn:${AWS::Partition}:s3:::netlumi-agent-monitoring-${NetlumiAccountId}/v1/${AWS::AccountId}.json' # The sync state, as tags on this function (Netlumi's scanner reads them). - Sid: SyncStateOnOwnFunction Effect: Allow Action: - 'lambda:ListTags' - 'lambda:TagResource' Resource: !Sub 'arn:${AWS::Partition}:lambda:${AWS::Region}:${AWS::AccountId}:function:netlumi-onboarding-setup-${AWS::AccountId}' # Netlumi-managed hardening: # the setup function's logs expire after 30 days (a log group that never # expires is flagged by posture tools). # The function and both custom resources depend on this group, so on deletion or rollback CloudFormation deletes it last, and # nothing can re-create it (see OwnLogGroup), so a retry after a failed # stack can create it again. NetlumiSetupLogGroup: Type: AWS::Logs::LogGroup Properties: LogGroupName: !Sub '/netlumi/onboarding-setup-${AWS::AccountId}' RetentionInDays: 30 # Never prints the event or its properties (the callback secret). NetlumiSetupFunction: Type: AWS::Lambda::Function DependsOn: NetlumiSetupLogGroup Properties: FunctionName: !Sub 'netlumi-onboarding-setup-${AWS::AccountId}' Description: Netlumi onboarding setup, status reporting and cleanup Runtime: python3.12 Handler: index.handler Role: !GetAtt NetlumiSetupLambdaRole.Arn # Netlumi-managed hardening: LoggingConfig: LogGroup: !Ref NetlumiSetupLogGroup # The custom resources' ServiceTimeout; the function answers before it ends. Timeout: 900 Code: ZipFile: | import calendar, concurrent.futures, json, re, threading, time, urllib.request, urllib.error import boto3 RULE = 'netlumi-cloudtrail-forwarder-{}' TRAIL = 'netlumi-trail' DNS_CONFIG = 'netlumi-dns-query-logging' DNS_LOG_GROUP = '/netlumi/route53resolver/query-logs' # --- generated: do not edit (forwarding spec and rules) --- # Per-request AI calls left out unless ForwardAIInvocationEvents is 'true'. # Must equal Netlumi's own list (a test compares them). HIGH_VOLUME_EVENTS = { "bedrock.amazonaws.com": [ "ApplyGuardrail", "Converse", "ConverseStream", "CountTokens", "GetAsyncInvoke", "InvokeAgent", "InvokeFlow", "InvokeInlineAgent", "InvokeModel", "InvokeModelWithBidirectionalStream", "InvokeModelWithResponseStream", "ListAsyncInvokes", "RenderPrompt", "Rerank", "Retrieve", "RetrieveAndGenerate", "RetrieveAndGenerateStream", "StartAsyncInvoke" ], "bedrock-agentcore.amazonaws.com": ["InvokeAgentRuntime", "InvokeBrowser", "InvokeCodeInterpreter"], "sagemaker.amazonaws.com": ["InvokeEndpoint", "InvokeEndpointAsync", "InvokeEndpointWithResponseStream"] } SPEC = { "write_verbs": [ "Create", "Delete", "Update", "Modify", "Put", "Attach", "Detach", "Associate", "Disassociate", "Tag", "Untag", "Run", "Launch", "Terminate", "Remove", "Deregister", "Stop", "Authorize", "Revoke", "Enable", "Disable", "Start", "Set", "Schedule", "Leave", "Deactivate", "Reset", "Copy", "Restore", "Export", "Replace", "Share", "Import", "Publish", "Send", "Register" ], "sources": [ "access-analyzer.amazonaws.com", "account.amazonaws.com", "acm.amazonaws.com", "aoss.amazonaws.com", "apigateway.amazonaws.com", "apprunner.amazonaws.com", "appsync.amazonaws.com", "athena.amazonaws.com", "autoscaling.amazonaws.com", "backup.amazonaws.com", "batch.amazonaws.com", "bedrock-agentcore.amazonaws.com", "bedrock.amazonaws.com", "cloudformation.amazonaws.com", "cloudfront.amazonaws.com", "cloudhsm.amazonaws.com", "cloudtrail.amazonaws.com", "codebuild.amazonaws.com", "codecommit.amazonaws.com", "codedeploy.amazonaws.com", "codepipeline.amazonaws.com", "cognito-identity.amazonaws.com", "cognito-idp.amazonaws.com", "comprehend.amazonaws.com", "config.amazonaws.com", "connect.amazonaws.com", "datasync.amazonaws.com", "dax.amazonaws.com", "dms.amazonaws.com", "ds.amazonaws.com", "dynamodb.amazonaws.com", "ec2-instance-connect.amazonaws.com", "ec2.amazonaws.com", "ecr.amazonaws.com", "ecs.amazonaws.com", "eks.amazonaws.com", "elasticache.amazonaws.com", "elasticbeanstalk.amazonaws.com", "elasticfilesystem.amazonaws.com", "elasticloadbalancing.amazonaws.com", "elasticmapreduce.amazonaws.com", "es.amazonaws.com", "events.amazonaws.com", "firehose.amazonaws.com", "fsx.amazonaws.com", "glue.amazonaws.com", "guardduty.amazonaws.com", "iam.amazonaws.com", "identitystore.amazonaws.com", "inspector2.amazonaws.com", "iot.amazonaws.com", "kafka.amazonaws.com", "kafkaconnect.amazonaws.com", "kendra.amazonaws.com", "kinesis.amazonaws.com", "kms.amazonaws.com", "lambda.amazonaws.com", "logs.amazonaws.com", "macie2.amazonaws.com", "mediaconvert.amazonaws.com", "medialive.amazonaws.com", "monitoring.amazonaws.com", "mq.amazonaws.com", "network-firewall.amazonaws.com", "organizations.amazonaws.com", "rds.amazonaws.com", "redshift-serverless.amazonaws.com", "redshift.amazonaws.com", "rekognition.amazonaws.com", "rolesanywhere.amazonaws.com", "route53.amazonaws.com", "route53domains.amazonaws.com", "route53resolver.amazonaws.com", "s3.amazonaws.com", "sagemaker.amazonaws.com", "secretsmanager.amazonaws.com", "securityhub.amazonaws.com", "securitylake.amazonaws.com", "servicecatalog.amazonaws.com", "ses.amazonaws.com", "shield.amazonaws.com", "signin.amazonaws.com", "sns.amazonaws.com", "sqs.amazonaws.com", "ssm.amazonaws.com", "sso-directory.amazonaws.com", "sso.amazonaws.com", "states.amazonaws.com", "storagegateway.amazonaws.com", "sts.amazonaws.com", "transfer.amazonaws.com", "wafv2.amazonaws.com", "workspaces.amazonaws.com" ], "extra_names": { "ec2.amazonaws.com": ["RequestSpotFleet", "RequestSpotInstances"], "ecs.amazonaws.com": ["ExecuteCommand"], "ses.amazonaws.com": ["VerifyDomainIdentity", "VerifyEmailIdentity"] }, "service_exceptions": { "ec2.amazonaws.com": ["RunInstances", "TerminateInstances"], "ecs.amazonaws.com": ["UpdateService"], "iam.amazonaws.com": ["CreateServiceLinkedRole"] }, "global_sources": [ "account.amazonaws.com", "cloudfront.amazonaws.com", "ecr-public.amazonaws.com", "iam.amazonaws.com", "organizations.amazonaws.com", "route53.amazonaws.com", "route53domains.amazonaws.com", "shield.amazonaws.com", "signin.amazonaws.com", "sts.amazonaws.com", "waf.amazonaws.com" ], "human_types": ["Root", "IAMUser", "FederatedUser", "SAMLUser", "WebIdentityUser", "IdentityCenterUser"], "long_term_key_prefix": "AKIA", "workload_reads": [ "AssumeRole", "AssumeRoleWithSAML", "AssumeRoleWithWebIdentity", "AssumeRoot", "BatchGetSecretValue", "GetAccountAuthorizationDetails", "GetCallerIdentity", "GetCredentialReport", "GetFederationToken", "GetFoundationModelAvailability", "GetModelInvocationLoggingConfiguration", "GetPasswordData", "GetRoleCredentials", "GetSecretValue", "GetSessionToken", "GetUseCaseForModelAccess", "ListFoundationModelAgreementOffers", "ListFoundationModels" ], "ssm_reads": ["GetParameter", "GetParameters", "GetParametersByPath"], "throttling_errors": [ "BandwidthLimitExceeded", "Client.RequestLimitExceeded", "EC2ThrottledException", "PriorRequestNotComplete", "ProvisionedThroughputExceededException", "RequestLimitExceeded", "RequestThrottled", "RequestThrottledException", "SlowDown", "ThrottledException", "Throttling", "ThrottlingException", "TooManyRequestsException", "TransactionInProgressException" ], "netlumi_roles": { "scanner": "NetlumiScannerRole", "forwarder": "NetlumiEventBridgeForwarderRole", "setup": "NetlumiSetupLambdaRole", "org_bootstrap": "NetlumiOrgBootstrapRole", "trail_logs": "NetlumiCloudTrailLogs", "trail_processor": "NetlumiCloudTrailProcessor", "trail_notification": "NetlumiTrailBucketNotifier" }, "signin_service_event_sources": ["signin.amazonaws.com", "sso.amazonaws.com"], "guardduty_global_resource_types": ["AccessKey"], "high_volume": HIGH_VOLUME_EVENTS } API = 'AWS API Call via CloudTrail' SIGNIN = 'AWS Console Sign In via CloudTrail' SERVICE_EVENT = 'AWS Service Event via CloudTrail' GUARDDUTY = 'GuardDuty Finding' # EventBridge's default "Event pattern size" quota (Service Quotas, adjustable). PATTERN_LIMIT = 2048 # Rule ids a region may hold; 'writes' keeps the name of the single rule of older stacks. WRITE_RULE_IDS = ('writes', 'writes-2', 'writes-3', 'writes-4') RULE_IDS = WRITE_RULE_IDS + ('reads', 'workload', 'failures', 'other') def compact(pattern): return json.dumps(pattern, separators=(',', ':')) def rule_suffix(rule_id): return '' if rule_id == 'writes' else '-' + rule_id def description(rule_id, scope): """The rule's description (at most 512 characters).""" if scope == 'signin': return 'Netlumi: only console sign-ins AWS records in this region (root, global sign-in). Nothing here is scanned.' kind = rule_id.split('-')[0] what = {'writes': 'write calls of the services Netlumi scans or detects threats on', 'reads': 'read-only calls by people, long-term keys and console sessions, root user activity', 'workload': 'selected reads by roles and workloads (role assumption, caller identity, secrets)', 'failures': 'failed calls (not throttling)', 'other': 'console and Identity Center sign-ins, GuardDuty findings'}[kind] if rule_id != kind: what += ' (part %s)' % rule_id.split('-')[1] if scope == 'global': return ('Netlumi: global-service events only (IAM, STS, CloudFront, Route 53...), delivered only in ' 'us-east-1; nothing here is scanned. Forwards ' + what + '.') return 'Netlumi real-time forwarder: ' + what + '.' # Writes, reads, workload and failures forward management # events only. Data and network activity events reach Netlumi only through # the agent rules, so a customer's own data-event trail is not forwarded (and # billed) by these rules. A missing eventCategory (event versions before # 1.07) still matches. NOT_DATA = [{'anything-but': ['Data', 'NetworkActivity']}, {'exists': False}] def _category(management_only): # The rules of older organisation member templates (already released) # have no eventCategory condition. return {'eventCategory': NOT_DATA} if management_only else {} def _verb_match(spec, name): return any(name.startswith(v) for v in spec['write_verbs']) def role_arn(partition, account, prefix): """The ARN of the Netlumi role named prefix-account (every role Netlumi's customer artefacts create is named so).""" return 'arn:%s:iam::%s:role/%s-%s' % (partition, account, prefix, account) def own_principals(roles): """Netlumi's own roles in this account (IAM role ARNs, exactly the roles the artefact creates) -> their sessions (wildcards on userIdentity.arn) and the role ARNs (exact, on requestParameters.roleArn). By exact name, never by prefix: a role anyone else names Netlumi... is forwarded.""" if not roles: raise ValueError('no Netlumi roles to exclude') sessions = [] for arn in roles: parts = arn.split(':', 5) if len(parts) != 6 or parts[0] != 'arn' or parts[2] != 'iam' or not parts[5].startswith('role/') or '*' in arn: raise ValueError('not an IAM role ARN: %s' % arn) # An assumed-role ARN names the role without its path. sessions.append('arn:%s:sts::%s:assumed-role/%s/*' % (parts[1], parts[4], parts[5].rsplit('/', 1)[1])) return {'sessions': sessions, 'roles': list(roles)} def _not_session(own): return [{'anything-but': {'wildcard': own['sessions']}}, {'exists': False}] def _caller(own): # Any caller but AWS services and Netlumi's own roles. return {'type': [{'anything-but': ['AWSService']}], 'arn': _not_session(own)} def _either(base, alternatives): """base AND any of alternatives. EventBridge refuses an $or with fewer than two objects, so a single alternative is merged into base.""" out = dict(base) if len(alternatives) == 1: out.update(alternatives[0]) else: out['$or'] = alternatives return out def _write_rule(spec, sources, management_only=True): # Every caller's writes but AWS services' (and the root user's, in reads), # Netlumi's own roles' included: they are rare (setup, # bootstrap), the customer sees them in the change log, and threat # detection skips them by exact name. names = [{'prefix': v} for v in spec['write_verbs']] names += sorted({n for s in sources for n in spec['extra_names'].get(s, [])}) main = {'eventSource': list(sources), 'eventName': names, 'userIdentity': {'type': [{'anything-but': ['AWSService', 'Root']}]}} exc = [{'eventSource': [s], 'eventName': spec['service_exceptions'][s], 'userIdentity': {'type': ['AWSService']}} for s in sources if s in spec['service_exceptions']] base = dict(_category(management_only), readOnly=[False, {'exists': False}], errorCode=[{'exists': False}]) detail = _either(base, [main] + exc) return {'detail-type': [API], 'detail': detail} def _write_rules(spec, sources, management_only=True): groups, cur = [], [] for s in sources: if cur and len(compact(_write_rule(spec, cur + [s], management_only))) > PATTERN_LIMIT: groups.append(cur) cur = [] cur = cur + [s] if cur: groups.append(cur) if len(groups) > len(WRITE_RULE_IDS): raise ValueError('write sources need %d rules; at most %d' % (len(groups), len(WRITE_RULE_IDS))) return [(WRITE_RULE_IDS[i], _write_rule(spec, g, management_only)) for i, g in enumerate(groups)] def _reads_rule(spec, glob, forward_ai, management_only=True): hv = spec['high_volume'] in_scope = [s for s in sorted(hv) if not glob or s in spec['global_sources']] # Every read by a long-term (AKIA) key, per-request AI calls included: # stolen long-term keys are how model abuse (LLMjacking) starts, and # applications rarely call models with them. long_term = {'readOnly': [True], 'userIdentity': {'accessKeyId': [{'prefix': spec['long_term_key_prefix']}]}} human = {'readOnly': [True], '$or': [ {'userIdentity': {'type': [t for t in spec['human_types'] if t != 'Root']}}, {'sessionCredentialFromConsole': ['true']}]} if in_scope and not forward_ai: # Other people's per-request AI calls stay out unless ForwardAIInvocationEvents. human['eventName'] = [{'anything-but': sorted({n for s in in_scope for n in hv[s]})}] ors = [{'userIdentity': {'type': ['Root']}}, long_term, human] if forward_ai and in_scope: # Invocations the write rules do not already forward (no verb prefix). names = sorted({n for s in in_scope for n in hv[s] if not _verb_match(spec, n)}) ors.append({'eventSource': in_scope, 'eventName': names}) detail = dict(_category(management_only), errorCode=[{'exists': False}]) detail['$or'] = ors if glob: detail['eventSource'] = spec['global_sources'] return {'detail-type': [API], 'detail': detail} def _workload_rule(spec, own, glob, management_only=True): """Workloads' reads stolen credentials are used for (split from reads, which it would push over the quota). Disjoint from reads: no root user, person, console session or long-term key, and none of these names is a per-request AI call.""" caller = {'type': [{'anything-but': ['AWSService'] + spec['human_types']}], 'arn': _not_session(own), 'accessKeyId': [{'anything-but': {'prefix': spec['long_term_key_prefix']}}, {'exists': False}]} workload = [{'eventName': spec['workload_reads'], 'requestParameters': {'roleArn': [{'anything-but': own['roles']}, {'exists': False}]}}] if not glob: workload.append({'eventSource': ['ssm.amazonaws.com'], 'eventName': spec['ssm_reads'], 'requestParameters': {'withDecryption': [True]}}) base = dict(_category(management_only), readOnly=[True], errorCode=[{'exists': False}], userIdentity=caller, sessionCredentialFromConsole=[{'anything-but': ['true']}, {'exists': False}]) detail = _either(base, workload) if glob: detail['eventSource'] = spec['global_sources'] return {'detail-type': [API], 'detail': detail} def _failures_rule(spec, own, glob, management_only=True): detail = dict(_category(management_only), errorCode=[{'anything-but': spec['throttling_errors']}], userIdentity=_caller(own)) if glob: detail['eventSource'] = spec['global_sources'] return {'detail-type': [API], 'detail': detail} def _other_rule(spec, glob): gd = {'source': ['aws.guardduty'], 'detail-type': [GUARDDUTY]} if glob: # The principal is global; other findings are about unselected resources. gd['detail'] = {'resource': {'resourceType': spec['guardduty_global_resource_types']}} return {'$or': [{'detail-type': [SIGNIN]}, {'detail-type': [SERVICE_EVENT], 'detail': {'eventSource': spec['signin_service_event_sources']}}, gd]} def rules(spec, scope, forward_ai, roles, management_only=True): """[(rule id, pattern)] of a region with this scope: 'regional' (a selected region), 'global' (us-east-1 when not selected: global-service events only) or 'signin' (console sign-ins only). roles: the ARNs of exactly the Netlumi roles this account holds (own_principals). management_only: the current templates' rules (False: the rules of older organisation member templates, without the eventCategory condition).""" own = own_principals(roles) if scope == 'signin': return [('other', {'detail-type': [SIGNIN]})] glob = scope == 'global' sources = [s for s in spec['sources'] if not glob or s in spec['global_sources']] out = _write_rules(spec, sources, management_only) out.append(('reads', _reads_rule(spec, glob, forward_ai, management_only))) out.append(('workload', _workload_rule(spec, own, glob, management_only))) out.append(('failures', _failures_rule(spec, own, glob, management_only))) out.append(('other', _other_rule(spec, glob))) for rid, p in out: if len(compact(p)) > PATTERN_LIMIT: raise ValueError('%s %s pattern is %d characters; the quota is %d' % (scope, rid, len(compact(p)), PATTERN_LIMIT)) return out # --- end of generated forwarding spec and rules --- # TemplateVersion values the code compares with: the current rule # layout starts at 9 (10 changed only the patterns), and 13 added the # workload rule and excludes Netlumi's roles by exact name. RULES_VERSION = 9 OWN_ROLES_VERSION = 13 def own_roles(props, account): """The ARNs of exactly the Netlumi roles this stack holds: the scanner, the forwarder, this function's role and, while the stack has it, the organisation bootstrap role. Their calls are not forwarded; a role that merely looks like them is.""" scanner = props.get('ScannerRoleArn') or '' partition = scanner.split(':')[1] if scanner.startswith('arn:') else 'aws' names = SPEC['netlumi_roles'] roles = [scanner or role_arn(partition, account, names['scanner']), role_arn(partition, account, names['forwarder'])] # From TemplateVersion 13 the setup role is named; a rollback to # older properties runs with a generated name, which is not excluded. if template_version(props) >= OWN_ROLES_VERSION: roles.append(role_arn(partition, account, names['setup'])) if props.get('OrgBootstrapRoleArn'): roles.append(props['OrgBootstrapRoleArn']) return roles def template_version(props): try: return int(str((props or {}).get('TemplateVersion') or '0').strip()) except ValueError: return 0 # The single rule per region of TemplateVersion 8 and 7, for a # rollback to them: they know only that rule name, so a rolled-back # update must leave only it. V8_DENIED = [{'prefix': 'AccessDenied'}, {'suffix': 'UnauthorizedOperation'}] def v8_pattern(props, scope): if scope == 'signin': return {'detail-type': [SIGNIN]} hv = {} if props.get('ForwardAIInvocationEvents') == 'true' else HIGH_VOLUME_EVENTS gs = SPEC['global_sources'] w, ro = {'readOnly': [False, {'exists': False}]}, {'readOnly': [True]} who = {'userIdentity': {'type': [{'anything-but': ['AWSService']}]}} gd = {'source': ['aws.guardduty'], 'detail-type': [GUARDDUTY]} if scope == 'global': g = {'eventSource': gs} lim = sorted(s for s in hv if s in gs) first = {'eventSource': [s for s in gs if s not in lim]} gd['detail'] = {'resource': {'resourceType': ['AccessKey']}} else: g, lim = {}, sorted(hv) first = {'eventSource': [{'anything-but': lim}]} if lim else {} def api(**d): return {'detail-type': [API], 'detail': d} ors = [api(**w, **first)] + [api(**w, eventSource=[s], eventName=[{'anything-but': hv[s]}]) for s in lim] ors += [api(**ro, **g, eventName=SPEC['workload_reads'], **who), api(**ro, **g, errorCode=V8_DENIED)] if not g: ors.append(api(**ro, **who, eventSource=['ssm.amazonaws.com'], eventName=SPEC['ssm_reads'], requestParameters={'withDecryption': [True]})) return {'$or': ors + [{'detail-type': [SIGNIN]}, gd]} def layout(props, scope, roles): """[(rule id, pattern)] the stack's properties ask for in a region (roles: own_roles).""" if template_version(props) < RULES_VERSION: return [('writes', v8_pattern(props, scope))] return rules(SPEC, scope, props.get('ForwardAIInvocationEvents') == 'true', roles) # Every request names its sender. Netlumi's callback is # behind Cloudflare, whose Browser Integrity Check refuses urllib's # default 'Python-urllib/3.x' with HTTP 403 (error code 1010). def user_agent(props): version = str((props or {}).get('TemplateVersion') or '').strip() or 'unknown' return 'netlumi-onboarding/%s (+https://netlumi.com)' % version _RESPOND_LOCK = threading.Lock() def respond(event, context, ok, data=None, reason='', pid=None): # Exactly one answer per request: the watchdog may race the setup. with _RESPOND_LOCK: if event.get('_netlumi_responded'): return event['_netlumi_responded'] = True body = json.dumps({ 'Status': 'SUCCESS' if ok else 'FAILED', 'Reason': (reason or 'See CloudWatch Logs: ' + context.log_stream_name)[:900], 'PhysicalResourceId': pid or event.get('PhysicalResourceId') or context.log_stream_name, 'StackId': event['StackId'], 'RequestId': event['RequestId'], 'LogicalResourceId': event['LogicalResourceId'], 'Data': data or {}, }).encode() req = urllib.request.Request(event['ResponseURL'], data=body, method='PUT', headers={'content-type': '', 'content-length': str(len(body)), 'User-Agent': user_agent(event.get('ResourceProperties'))}) urllib.request.urlopen(req, timeout=30) def notify(props, event_name, timeout=60, **fields): payload = {'registrationId': props['RegistrationId'], 'callbackSecret': props['CallbackSecret'], 'event': event_name, 'templateVersion': props.get('TemplateVersion', '')} payload.update(fields) req = urllib.request.Request(props['CallbackUrl'], data=json.dumps(payload).encode(), headers={'Content-Type': 'application/json', 'User-Agent': user_agent(props)}, method='POST') try: with urllib.request.urlopen(req, timeout=timeout) as resp: return True, json.loads(resp.read() or b'{}') except urllib.error.HTTPError as e: try: return False, json.loads(e.read() or b'{}') except ValueError: return False, {'message': 'HTTP %d' % e.code} except Exception as e: return False, {'message': 'Netlumi could not be reached: %s' % type(e).__name__} def stack_status(stack_id): cfn = boto3.client('cloudformation') return cfn.describe_stacks(StackName=stack_id)['Stacks'][0]['StackStatus'] def first_failure(stack_id): cfn = boto3.client('cloudformation') events = cfn.describe_stack_events(StackName=stack_id)['StackEvents'] failed = [e for e in events if e.get('ResourceStatus', '').endswith('_FAILED') and 'cancelled' not in e.get('ResourceStatusReason', '') and e.get('ResourceType') != 'AWS::CloudFormation::Stack'] if not failed: return 'The stack rolled back.' e = failed[-1] return '%s %s: %s' % (e['ResourceStatus'], e['LogicalResourceId'], e.get('ResourceStatusReason', '')) def rule_plan(props): p = props or {} if p.get('RealtimeForwarding') == 'disabled': # Real-time is not offered: no forwarding rule anywhere. return {} plan = {r: 'regional' for r in regions_of(p)} for key, scope in (('GlobalEventsRegion', 'global'), ('SignInEventsRegions', 'signin')): for r in regions_of(p, key): plan.setdefault(r, scope) return plan def error_code(e): return ((getattr(e, 'response', None) or {}).get('Error') or {}).get('Code') or type(e).__name__ # Codes AWS gives while a new account (or a region in it) is still being # activated, and for a refused call (the reason names a policy only # when AWS's message does). ACTIVATING_CODES = ('OptInRequired', 'SubscriptionRequiredException') DENIED_CODES = ('AccessDenied', 'AccessDeniedException', 'UnauthorizedOperation', 'AuthorizationError') def failure_of(region, scope, e): code = error_code(e) low = (code + ' ' + str(e)).lower() if code in ('UnrecognizedClientException', 'InvalidClientTokenId', 'AuthFailure') \ or 'security token included in the request is invalid' in low: kind, why = 'region_not_enabled', 'The region is not enabled in this AWS account (opt-in region).' elif code in ACTIVATING_CODES: kind, why = 'account_activating', ('AWS account still activating (%s): a new AWS account can take up to a day ' 'before every region and service is ready. Update the stack later to turn ' 'real time on here.' % code) elif code in DENIED_CODES or 'accessdenied' in code.lower() or 'not authorized' in low: if 'service control policy' in low: kind, why = 'access_denied', 'Blocked by a service control policy (%s).' % code elif 'permissions boundary' in low: kind, why = 'access_denied', 'Blocked by a permissions boundary (%s).' % code else: kind, why = 'refused', ('AWS refused the call (%s). New AWS accounts can take a while to activate a region; ' 'otherwise an SCP or permissions boundary is blocking it.' % code) elif code == 'LimitExceededException': kind, why = 'error', ('An EventBridge quota of the default event bus is reached here (300 rules, ' 'or 30 rules with wildcard patterns): %s' % str(e)) else: kind, why = 'error', '%s: %s' % (code, str(e)) return {'region': region, 'scope': scope, 'code': kind, 'reason': why[:300]} RETRY_SECONDS = 8 # Regions are set up in parallel, at most this many at once. PARALLEL_REGIONS = 8 # Kept back from the Lambda's remaining time to report: the completed callback # (60 s) and the CloudFormation response (30 s), with a margin. RESERVE_SECONDS = 100 # The watchdog answers CloudFormation this long before the Lambda would time # out, if nothing else has. WATCHDOG_SECONDS = 20 def remaining_seconds(context): try: return context.get_remaining_time_in_millis() / 1000.0 except Exception: return 900.0 def deadline_of(context): return time.time() + remaining_seconds(context) - RESERVE_SECONDS def in_regions(work, regions, deadline): """{region: (result, exception)} of work(region) for each region, in parallel. A region without an answer by the deadline is missing.""" regions = sorted(set(regions)) if not regions: return {} pool = concurrent.futures.ThreadPoolExecutor(max_workers=min(PARALLEL_REGIONS, len(regions))) futures = {pool.submit(work, r): r for r in regions} done, _ = concurrent.futures.wait(futures, timeout=max(0, deadline - time.time())) pool.shutdown(wait=False) out = {} for f in done: try: out[futures[f]] = (f.result(), None) except Exception as e: out[futures[f]] = (None, e) return out # Regions with Netlumi's receiving bus; elsewhere the us-east-1 bus. INGRESS_REGIONS = set(( 'us-east-2 us-west-1 us-west-2 eu-west-1 eu-west-2 eu-west-3 eu-central-1 eu-north-1 ' 'ap-southeast-1 ap-southeast-2 ap-northeast-1 ap-northeast-2 ap-northeast-3 ap-south-1 ' 'ca-central-1 sa-east-1').split()) def bus_for(region, netlumi): if region in INGRESS_REGIONS: return 'arn:aws:events:%s:%s:event-bus/netlumi-realtime-ingress' % (region, netlumi) return 'arn:aws:events:us-east-1:%s:event-bus/netlumi-realtime-events' % netlumi def probe(ev, account, region, bus): # Can this account PutEvents here (own bus, then Netlumi's)? An SCP # or boundary deny blocks delivery. Before binding Netlumi's bus # does not admit the account yet (implicit deny: expected). for target in ('arn:aws:events:%s:%s:event-bus/default' % (region, account), bus): try: out = ev.put_events(Entries=[{'Source': 'netlumi.probe', 'DetailType': 'Netlumi delivery probe', 'Detail': '{}', 'EventBusName': target}]) e = (out.get('Entries') or [{}])[0] code, msg = e.get('ErrorCode') or '', e.get('ErrorMessage') or '' except Exception as x: code = error_code(x) msg = str(x) low = (code + ' ' + msg).lower() if 'explicit deny' in low or 'service control policy' in low or 'permissions boundary' in low: return 'delivery_denied', 'Your organization\'s policy (%s) denies events:PutEvents %s, so no event reaches Netlumi.' % ( 'SCP' if 'service control' in low else 'IAM', 'to us-east-1' if target == bus and region not in bus else 'here') if 'resourcenotfound' in low and target == bus: return 'error', 'Netlumi has no receiving event bus in this region yet.' if code: print('probe %s: %s' % (region, code)) return None def put_rule(ev, name, rule, target): ev.put_rule(Name=name, **rule) ev.put_targets(Rule=name, Targets=[target]) def put_region(ev, account, props, region, scope): """None when the region forwards, else its failure.""" role = 'arn:aws:iam::%s:role/NetlumiEventBridgeForwarderRole-%s' % (account, account) bus = bus_for(region, props['NetlumiAccountId']) target = {'Id': 'NetlumiEventBus', 'Arn': bus, 'RoleArn': role} want = layout(props, scope, own_roles(props, account)) for rid, pat in want: # Read-only events reach a rule only in this state (Netlumi's bus rules too). rule = dict(State='ENABLED_WITH_ALL_CLOUDTRAIL_MANAGEMENT_EVENTS', Description=description(rid, scope), EventPattern=compact(pat)) for attempt in (1, 2): try: put_rule(ev, RULE.format(account) + rule_suffix(rid), rule, target) except Exception as e: f = failure_of(region, scope, e) # A refusal can be IAM propagation (PassRole): retry once. if attempt == 1 and f['code'] == 'refused': time.sleep(RETRY_SECONDS) continue # Rules already put stay: an update puts the rest. print('rule %s %s (%s): %s' % (region, rid, scope, type(e).__name__)) return f break # Rules of another layout (scope change, template version) go. remove_rules(ev, account, region, [r for r in RULE_IDS if r not in dict(want)]) # The rules stay: once the policy allows it, delivery starts. why = probe(ev, account, region, bus) if why: return {'region': region, 'scope': scope, 'code': why[0], 'reason': why[1]} return None # Opt-in states of a region the account can use. ENABLED_REGION_STATES = ('opt-in-not-required', 'opted-in') NOT_ENABLED_REASON = ('The region is not enabled in this AWS account (opt-in region). Enable it ' '(AWS console: Account > AWS Regions), then update the stack.') def region_states(): """{region: OptInStatus} of every region (ec2:DescribeRegions AllRegions), or None when it cannot be read: the precheck is then skipped and EventBridge's answer classifies the region.""" try: out = boto3.client('ec2').describe_regions(AllRegions=True) except Exception as e: print('describe regions: %s' % error_code(e)) return None states = {r['RegionName']: r.get('OptInStatus', '') for r in out.get('Regions') or [] if r.get('RegionName')} return states or None def not_enabled(regions, states): """The regions the account has not enabled (sorted); none when unknown.""" if not states: return [] return sorted(r for r in set(regions) if states.get(r) not in ENABLED_REGION_STATES) def put_rules(account, props, plan, deadline, states=None): # A region the account has not enabled gets no EventBridge call: # it is reported as such at once, not after a timeout and retry. off = set(not_enabled(plan, states)) work = {r: s for r, s in plan.items() if r not in off} clients = {r: boto3.client('events', region_name=r) for r in sorted(work)} results = in_regions(lambda r: put_region(clients[r], account, props, r, work[r]), work, deadline) ok, failed = set(), [] for region in sorted(plan): if region in off: failed.append({'region': region, 'scope': plan[region], 'code': 'region_not_enabled', 'reason': NOT_ENABLED_REASON}) continue if region not in results: print('rule %s: no answer before the deadline' % region) failed.append({'region': region, 'scope': plan[region], 'code': 'error', 'reason': 'EventBridge did not answer in this region before the setup had to report. ' 'Update the stack to retry.'}) continue fail, exc = results[region] if exc is not None: fail = failure_of(region, plan[region], exc) if fail: failed.append(fail) else: ok.add(region) return [r for r in regions_of(props) if r in ok], failed def remove_rules(ev, account, region, rule_ids): for rid in rule_ids: name = RULE.format(account) + rule_suffix(rid) try: ev.remove_targets(Rule=name, Ids=['NetlumiEventBus']) ev.delete_rule(Name=name) except Exception as e: if 'ResourceNotFound' not in str(e): print('cleanup %s %s: %s' % (region, rid, type(e).__name__)) def delete_rules(account, regions, deadline): clients = {r: boto3.client('events', region_name=r) for r in sorted(regions)} for region, (_, e) in in_regions(lambda r: remove_rules(clients[r], account, r, RULE_IDS), regions, deadline).items(): if e is not None: print('cleanup %s: %s' % (region, type(e).__name__)) def rules_data(deployed, failed, warnings=()): # Every success response carries all three: outputs read RegionsFailed and Warnings. what = {'global': ' (global-service events)', 'signin': ' (console sign-ins)'} warn = '; '.join('%s%s: %s' % (f['region'], what.get(f['scope'], ''), f['reason']) for f in failed) parts = '; '.join(w['message'] for w in warnings) return {'RegionsDeployed': ','.join(deployed), 'RegionsFailed': warn[:900] or 'none', 'Warnings': parts[:900] or 'none'} # Optional parts (CloudTrail, DNS query logging, EKS access) never fail the # stack: a failure is a warning, reported to Netlumi with the # completed callback and in the stack's SetupWarnings output. def warning(part, e, what): msg = str(e).replace('\n', ' ') return {'part': part, 'code': error_code(e)[:64], 'message': ('%s: %s (%s). The account is connected without it; fix the cause and update the stack.' % (what, msg[:200], error_code(e)))[:400]} # The PhysicalResourceId suffix of a setup resource whose stack created # netlumi-trail: only then does deleting the stack delete the trail (the # template's condition only says whether a trail was asked for). TRAIL_OWNED = '-trail' # CloudTrail can refuse a new trail until its bucket policy propagates. TRAIL_RETRY_SECONDS = 5 def list_trails(): try: return boto3.client('cloudtrail').describe_trails(includeShadowTrails=True).get('trailList') or [] except Exception as e: print('describe trails: %s' % error_code(e)) return None def records_write_management(sel): """Whether a trail's event selectors include write management events (the same rule as for an organization trail). No selectors is the default: all management events.""" advanced = sel.get('AdvancedEventSelectors') or [] if advanced: for a in advanced: management = read_only = False for f in a.get('FieldSelectors') or []: if f.get('Field') == 'eventCategory' and 'Management' in (f.get('Equals') or []): management = True if f.get('Field') == 'readOnly' and 'true' in (f.get('Equals') or []): read_only = True if management and not read_only: return True return False basic = sel.get('EventSelectors') or [] if not basic: return True for b in basic: if b.get('IncludeManagementEvents') is False or b.get('ReadWriteType') == 'ReadOnly': continue return True return False def trail_name(t): return t.get('Name') or (t.get('TrailARN') or '').split('/')[-1] def excludes_management(ct, t): """A warning when the reused trail does not log write management events (real time then receives nothing); None when it does or cannot be read. Netlumi never edits a customer's trail.""" try: sel = ct.get_event_selectors(TrailName=t.get('TrailARN') or trail_name(t)) except Exception as e: print('event selectors: %s' % error_code(e)) return None if records_write_management(sel): return None return {'part': 'cloudtrail', 'code': 'trail_excludes_management', 'message': ('Real-time changes need management events: trail %s does not log write management ' 'events, so real time receives no changes. Add management events (write or all) to ' 'trail %s; Netlumi does not change your trail.' % (trail_name(t), trail_name(t)))[:400]} def ensure_trail(bucket, deadline, trails=None): """(mode, trail name, warnings). The same rule as for an organization: a trail that covers every region, logs and records write management events feeds real time, and Netlumi then creates none. organization: an organization trail covers every region. It counts without its status (a member account may not read it; only a status that says it is stopped rules it out), and netlumi-trail is never created next to it; existing: another multi-region trail of this account is logging; ours: netlumi-trail of this stack is logging (or restarted); created: none of these (single-region trails do not count).""" ct = boto3.client('cloudtrail') if trails is None: trails = ct.describe_trails(includeShadowTrails=True)['trailList'] # netlumi-agent-activity logs data events only: never the # trail real time uses, and never a reason to warn. trails = [t for t in trails if trail_name(t) != AGENT_TRAIL] for t in trails: if t.get('IsOrganizationTrail') and t.get('IsMultiRegionTrail'): try: if ct.get_trail_status(Name=t.get('TrailARN') or trail_name(t)).get('IsLogging') is False: # Readable and stopped: it feeds nothing. print('organization trail %s is not logging' % trail_name(t)) continue except Exception as e: print('organization trail status: %s' % error_code(e)) w = excludes_management(ct, t) return 'organization', trail_name(t), [w] if w else [] for t in trails: if not t.get('IsMultiRegionTrail') or t.get('IsOrganizationTrail'): continue try: logging = ct.get_trail_status(Name=t.get('TrailARN') or trail_name(t)).get('IsLogging') except Exception as e: print('trail status %s: %s' % (t.get('Name'), error_code(e))) continue if not logging: continue if t.get('Name') == TRAIL and bucket and t.get('S3BucketName') == bucket: return 'ours', TRAIL, [] w = excludes_management(ct, t) return 'existing', trail_name(t), [w] if w else [] for t in trails: if t.get('Name') == TRAIL and bucket and t.get('S3BucketName') == bucket: # This stack's trail, stopped: start it again. ct.start_logging(Name=TRAIL) return 'ours', TRAIL, [] while True: try: ct.create_trail(Name=TRAIL, S3BucketName=bucket, IsMultiRegionTrail=True, IncludeGlobalServiceEvents=True, EnableLogFileValidation=True) break except Exception as e: if error_code(e) == 'InsufficientS3BucketPolicyException' and time.time() + TRAIL_RETRY_SECONDS < deadline: time.sleep(TRAIL_RETRY_SECONDS) continue raise ct.start_logging(Name=TRAIL) return 'created', TRAIL, [] def trail_warning(e): """The warning of a trail Netlumi could not set up (it names the quota and an organization's denial).""" code = error_code(e) low = (code + ' ' + str(e)).lower() if code == 'MaximumNumberOfTrailsExceededException': return {'part': 'cloudtrail', 'code': code, 'message': ('This region already has 5 trails (the AWS limit), so Netlumi could not create ' 'netlumi-trail. Real time needs one multi-region trail that logs management events: ' 'make one of yours multi-region (Netlumi then uses it) or remove one, then update the stack.')} if code in DENIED_CODES and ('service control policy' in low or 'explicit deny' in low): return {'part': 'cloudtrail', 'code': 'trail_denied', 'message': ('Your organization\'s policy denies creating a CloudTrail trail (%s). Real time needs ' 'a multi-region trail that logs management events: ask your cloud team for one, then ' 'update the stack.' % code)} return warning('cloudtrail', e, 'CloudTrail trail (needed for real-time changes)') # Accounts enrolled in AWS Control Tower: its execution role, # read with iam:GetRole on exactly that role. Netlumi never changes it. CONTROL_TOWER_ROLE = 'AWSControlTowerExecution' def governance(trails): """'control_tower' when AWS Control Tower manages the account, else ''.""" try: if boto3.client('iam').get_role(RoleName=CONTROL_TOWER_ROLE).get('Role'): return 'control_tower' except Exception as e: if error_code(e) != 'NoSuchEntity': print('governance: %s' % error_code(e)) for t in trails or []: if t.get('IsOrganizationTrail') and str(t.get('Name', '')).startswith('aws-controltower-'): return 'control_tower' return '' def delete_trail(): ct = boto3.client('cloudtrail') try: ct.delete_trail(Name=TRAIL) except Exception as e: if 'TrailNotFound' not in str(e): print('trail cleanup: %s' % type(e).__name__) def dns_vpcs(props): if (props or {}).get('EnableDnsQueryLogging') != 'true': return [] v = props.get('DnsQueryLoggingVpcIds') or [] if isinstance(v, str): v = v.split(',') return sorted({x.strip() for x in v if x.strip()}) def find_dns_config(r53r): out = r53r.list_resolver_query_log_configs(Filters=[{'Name': 'Name', 'Values': [DNS_CONFIG]}]) for c in out.get('ResolverQueryLogConfigs', []): if c.get('Name') == DNS_CONFIG: return c return None def dns_associations(r53r, cfg_id): out = r53r.list_resolver_query_log_config_associations( Filters=[{'Name': 'ResolverQueryLogConfigId', 'Values': [cfg_id]}]) return [a for a in out.get('ResolverQueryLogConfigAssociations', []) if a.get('Status') not in ('DELETING', 'FAILED')] def ensure_dns(account, region, vpcs): logs = boto3.client('logs', region_name=region) try: logs.create_log_group(logGroupName=DNS_LOG_GROUP) except Exception as e: if 'ResourceAlreadyExists' not in type(e).__name__ + str(e): raise logs.put_retention_policy(logGroupName=DNS_LOG_GROUP, retentionInDays=30) r53r = boto3.client('route53resolver', region_name=region) cfg = find_dns_config(r53r) if cfg is None: cfg = r53r.create_resolver_query_log_config( Name=DNS_CONFIG, CreatorRequestId='netlumi-%s-%s' % (account, region), DestinationArn='arn:aws:logs:%s:%s:log-group:%s' % (region, account, DNS_LOG_GROUP), )['ResolverQueryLogConfig'] have = {a['ResourceId'] for a in dns_associations(r53r, cfg['Id'])} for vpc in vpcs: if vpc not in have: r53r.associate_resolver_query_log_config(ResolverQueryLogConfigId=cfg['Id'], ResourceId=vpc) DNS_POLL_SECONDS = 5 def remove_dns(region, vpcs=None, deadline=None): # vpcs=None: remove all and the config (the log group is kept). r53r = boto3.client('route53resolver', region_name=region) cfg = find_dns_config(r53r) if cfg is None: return for a in dns_associations(r53r, cfg['Id']): if vpcs is None or a['ResourceId'] in vpcs: r53r.disassociate_resolver_query_log_config( ResolverQueryLogConfigId=cfg['Id'], ResourceId=a['ResourceId']) if vpcs is not None: return end = deadline if deadline is not None else time.time() + 300 while dns_associations(r53r, cfg['Id']) and time.time() + DNS_POLL_SECONDS < end: time.sleep(DNS_POLL_SECONDS) r53r.delete_resolver_query_log_config(ResolverQueryLogConfigId=cfg['Id']) def sync_dns(account, props, old=None, deadline=None): """Warnings; DNS query logging is optional.""" want, had = dns_vpcs(props), dns_vpcs(old) if not want and not had: return [] warnings, placed = [], {} if want: found = {} for region in regions_of(props): try: ec2 = boto3.client('ec2', region_name=region) for v in ec2.describe_vpcs(Filters=[{'Name': 'vpc-id', 'Values': want}]).get('Vpcs', []): found.setdefault(v['VpcId'], region) except Exception as e: warnings.append(warning('dns', e, 'DNS query logging: could not list the VPCs in %s' % region)) missing = [v for v in want if v not in found] if missing: warnings.append({'part': 'dns', 'code': 'VpcNotFound', 'message': ('DNS query logging: VPCs not found in the selected regions: %s. Check the IDs ' 'and regions, then update the stack.' % ', '.join(missing))[:400]}) for vpc, region in found.items(): placed.setdefault(region, []).append(vpc) dropped = set(had) - set(want) for region in sorted(set(regions_of(props)) | set(regions_of(old or {}))): try: if region in placed: ensure_dns(account, region, sorted(placed[region])) if dropped: remove_dns(region, dropped) else: remove_dns(region, deadline=deadline) except Exception as e: warnings.append(warning('dns', e, 'DNS query logging in %s' % region)) return warnings def eks_clusters(props): v = (props or {}).get('EksClusters') or [] if isinstance(v, str): v = v.split(',') return sorted({x.strip() for x in v if x.strip()}) def eks_view_policy(partition): return 'arn:%s:eks::aws:cluster-access-policy/AmazonEKSViewPolicy' % partition def sync_eks(region, partition, props, old=None): """Warnings. The scanner role reads each listed cluster with AmazonEKSViewPolicy (never Secrets); optional.""" role = props.get('ScannerRoleArn') or '' want, had = eks_clusters(props), eks_clusters(old) if not want and not had: return [] eks, warnings = boto3.client('eks', region_name=region), [] for cluster in want: try: try: eks.create_access_entry(clusterName=cluster, principalArn=role, type='STANDARD') except Exception as e: if error_code(e) != 'ResourceInUseException': raise eks.associate_access_policy(clusterName=cluster, principalArn=role, policyArn=eks_view_policy(partition), accessScope={'type': 'cluster'}) except Exception as e: warnings.append(warning('eks', e, 'EKS access to cluster %s in %s' % (cluster, region))) for cluster in sorted(set(had) - set(want)): remove_eks(eks, cluster, role) return warnings def remove_eks(eks, cluster, role): try: eks.delete_access_entry(clusterName=cluster, principalArn=role) except Exception as e: if error_code(e) != 'ResourceNotFoundException': print('eks cleanup %s: %s' % (cluster, type(e).__name__)) # Stacks of TemplateVersion 11 and earlier created the EKS access entries # as stack resources. An update from them removes those resources, and # CloudFormation deletes them in its cleanup, after this function # answered: the entries are put again once the update is complete. EKS_POLL_SECONDS = 5 def resync_eks_after_cleanup(stack_id, region, partition, props, deadline): while time.time() + EKS_POLL_SECONDS < deadline: status = stack_status(stack_id) if status == 'UPDATE_COMPLETE': for w in sync_eks(region, partition, props): print('eks after update: %s' % w['code']) return if status not in ('UPDATE_IN_PROGRESS', 'UPDATE_COMPLETE_CLEANUP_IN_PROGRESS'): return time.sleep(EKS_POLL_SECONDS) def regions_of(props, key='Regions'): r = props.get(key) or [] if isinstance(r, str): r = r.split(',') return [x.strip() for x in r if x.strip()] BOOTSTRAP_WINDOW_SECONDS = 7 * 24 * 3600 def bootstrap_not_after(now=None): t = time.time() if now is None else now return time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime(t + BOOTSTRAP_WINDOW_SECONDS)) # --- AI agent monitoring --- # Every 15 minutes (and on create and update) this function reads the list of # agent tool roles Netlumi publishes for this account, validates it here and # applies it: one data-events-only trail, netlumi-agent-activity, whose # selectors name only those roles' sessions, and -agents forwarding rules. # Netlumi has no write permission in the account; a list that fails any check # is refused and the last good one stays. The state is kept as tags on this # function (netlumi:agent-*), which Netlumi's scanner reads. AGENT_TRAIL = 'netlumi-agent-activity' AGENT_RULE_IDS = ('agents', 'agents-2', 'agents-3', 'agents-4') AGENT_LIST_BUCKET = 'netlumi-agent-monitoring-{}' AGENT_LIST_KEY = 'v1/{}.json' AGENT_VALUE_LIMIT = 500 AGENT_TYPES = ('AWS::Bedrock::AgentAlias', 'AWS::Bedrock::KnowledgeBase', 'AWS::BedrockAgentCore::Gateway', 'AWS::DynamoDB::Table', 'AWS::Lambda::Function', 'AWS::S3::Object', 'AWS::SES::EmailIdentity', 'AWS::SNS::Topic', 'AWS::SQS::Queue') AGENT_MODEL_SOURCES = ['bedrock.amazonaws.com', 'bedrock-agentcore.amazonaws.com'] AGENT_MODEL_NAMES = ['Converse', 'ConverseStream', 'InvokeAgentRuntime', 'InvokeModel', 'InvokeModelWithResponseStream'] # What one forwarded agent event can cost the account, worst case: # CloudTrail data event $0.10 per 100,000, EventBridge delivery $1 per # million, and EventBridge ingestion of a data event counted at $1 per million. AGENT_EVENT_USD = 3.0 / 1000000 TAG_STATE, TAG_VERSION, TAG_REASON, TAG_CAPPED, TAG_RULES = ( 'netlumi:agent-monitoring', 'netlumi:agent-list-version', 'netlumi:agent-reason', 'netlumi:agent-capped-month', 'netlumi:agent-rules') AGENT_ROLE_NAME = re.compile(r'^[\w+=,.@-]{1,64}$') AGENT_REGION = re.compile(r'^[a-z]{2}(-[a-z]+)+-\d$') def agent_prefix(partition, account, role): # The trailing slash keeps ledger-tools from matching ledger-tools-admin. return 'arn:%s:sts::%s:assumed-role/%s/' % (partition, account, role) def agent_monitored(lst): return [i for i in lst.get('identities') or [] if i.get('monitor')] def agent_selectors(lst, account, partition): """(advanced event selectors, value count) for the list. Must equal Netlumi's own copy of this function (a test compares them).""" ids = agent_monitored(lst) types = set(lst.get('resource_types') or []) every, s3_readers, ddb_readers, s3_scope, ddb_scope = [], [], [], set(), set() for i in ids: p = agent_prefix(partition, account, i['role_name']) every.append(p) s3 = [r for r in i.get('read_scope') or [] if ':s3:::' in r] ddb = [r for r in i.get('read_scope') or [] if ':s3:::' not in r and ':dynamodb:' in r and ':table/' in r] s3_scope.update(s3) ddb_scope.update(ddb) if s3: s3_readers.append(p) if ddb: ddb_readers.append(p) out = [] if not every: return out, 0 data = {'Field': 'eventCategory', 'Equals': ['Data']} def rtype(t): return {'Field': 'resources.type', 'Equals': [t]} def who(p): return {'Field': 'userIdentity.arn', 'StartsWith': list(p)} def names(*n): return {'Field': 'eventName', 'Equals': list(n)} writes = {'Field': 'readOnly', 'Equals': ['false']} def add(name, *fields): out.append({'Name': name, 'FieldSelectors': [dict(f) for f in fields]}) if 'AWS::S3::Object' in types: add('netlumi-agents-s3-writes', data, rtype('AWS::S3::Object'), writes, who(every)) if s3_scope: add('netlumi-agents-s3-reads', data, rtype('AWS::S3::Object'), names('GetObject'), {'Field': 'resources.ARN', 'StartsWith': sorted(s3_scope)}, who(s3_readers)) if 'AWS::DynamoDB::Table' in types: add('netlumi-agents-dynamodb-writes', data, rtype('AWS::DynamoDB::Table'), writes, who(every)) if ddb_scope: add('netlumi-agents-dynamodb-reads', data, rtype('AWS::DynamoDB::Table'), names('BatchGetItem', 'GetItem', 'Query', 'Scan'), {'Field': 'resources.ARN', 'Equals': sorted(ddb_scope)}, who(ddb_readers)) if 'AWS::Lambda::Function' in types: add('netlumi-agents-lambda', data, rtype('AWS::Lambda::Function'), names('Invoke'), who(every)) if 'AWS::BedrockAgentCore::Gateway' in types: add('netlumi-agents-agentcore-gateway', data, rtype('AWS::BedrockAgentCore::Gateway'), who(every)) if 'AWS::SES::EmailIdentity' in types: add('netlumi-agents-ses', data, rtype('AWS::SES::EmailIdentity'), who(every)) if 'AWS::SNS::Topic' in types: add('netlumi-agents-sns', data, rtype('AWS::SNS::Topic'), names('Publish', 'PublishBatch'), who(every)) if 'AWS::SQS::Queue' in types: add('netlumi-agents-sqs', data, rtype('AWS::SQS::Queue'), names('SendMessage', 'SendMessageBatch'), who(every)) if 'AWS::Bedrock::AgentAlias' in types: add('netlumi-agents-bedrock-agent-alias', data, rtype('AWS::Bedrock::AgentAlias'), who(every)) if 'AWS::Bedrock::KnowledgeBase' in types: add('netlumi-agents-bedrock-knowledge-base', data, rtype('AWS::Bedrock::KnowledgeBase'), who(every)) values = sum(len(f.get('Equals') or []) + len(f.get('StartsWith') or []) for s in out for f in s['FieldSelectors']) return out, values def agent_rule_pattern(ids, account, partition): pc = [{'prefix': agent_prefix(partition, account, i['role_name'])} for i in ids] mc = [{'prefix': agent_prefix(partition, account, i['role_name'])} for i in ids if i.get('model_calls')] data = {'eventCategory': ['Data'], 'userIdentity': {'arn': pc}} detail = data if mc: detail = {'$or': [data, {'eventCategory': ['Management'], 'eventSource': AGENT_MODEL_SOURCES, 'eventName': AGENT_MODEL_NAMES, 'userIdentity': {'arn': mc}}]} return compact({'detail-type': [API], 'detail': detail}) def agent_rule_patterns(lst, account, partition): """The -agents rules' patterns: the monitored identities in order, split greedily within PATTERN_LIMIT, at most len(AGENT_RULE_IDS) rules. Must equal the engine's AgentRulePatterns.""" out, cur = [], [] for i in agent_monitored(lst): nxt = cur + [i] if cur and len(agent_rule_pattern(nxt, account, partition)) > PATTERN_LIMIT: out.append(agent_rule_pattern(cur, account, partition)) cur = [i] else: cur = nxt if len(cur) == 1 and len(agent_rule_pattern(cur, account, partition)) > PATTERN_LIMIT: raise ValueError('role %s alone exceeds the pattern size' % i['role_name']) if cur: out.append(agent_rule_pattern(cur, account, partition)) if len(out) > len(AGENT_RULE_IDS): raise ValueError('the agent roles need %d forwarding rules; at most %d' % (len(out), len(AGENT_RULE_IDS))) return out def agent_list_problem(lst, account, partition, cap, applied, role_exists): """None when the list may be applied, else the reason it is refused role_exists(name) -> bool.""" if not isinstance(lst, dict) or lst.get('schema') != 1: return 'schema' if str(lst.get('account')) != account: return 'account' version = lst.get('version') if not isinstance(version, int) or isinstance(version, bool) or version < applied: return 'version' ids = lst.get('identities') if not isinstance(ids, list): return 'identities' s3_arn = re.compile(r'^arn:%s:s3:::[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]/$' % re.escape(partition)) ddb_arn = re.compile(r'^arn:%s:dynamodb:[a-z0-9-]+:%s:table/[A-Za-z0-9_.-]{3,255}$' % (re.escape(partition), account)) seen = set() for i in ids: if not isinstance(i, dict) or not AGENT_ROLE_NAME.match(str(i.get('role_name') or '')): return 'role_name' if i['role_name'] in seen: return 'duplicate_role' seen.add(i['role_name']) for r in i.get('read_scope') or []: if not isinstance(r, str) or not (s3_arn.match(r) or ddb_arn.match(r)): return 'read_scope' for r in i.get('regions') or []: if not isinstance(r, str) or not AGENT_REGION.match(r): return 'regions' if any(t not in AGENT_TYPES for t in lst.get('resource_types') or []): return 'resource_type' _, values = agent_selectors(lst, account, partition) if values > AGENT_VALUE_LIMIT: return 'value_limit' try: agent_rule_patterns(lst, account, partition) except ValueError: return 'pattern_size' try: usd = float((lst.get('estimate') or {}).get('usd_month') or 0) except (TypeError, ValueError): return 'estimate' if usd > cap: return 'over_cap' for i in agent_monitored(lst): if not role_exists(i['role_name']): return 'unknown_role' return None def agent_role_exists(name): try: boto3.client('iam').get_role(RoleName=name) return True except Exception as e: print('agent role %s: %s' % (name, error_code(e))) return False def read_agent_list(netlumi, account): """The published list, or None when there is none yet.""" s3 = boto3.client('s3', region_name='us-east-1') try: out = s3.get_object(Bucket=AGENT_LIST_BUCKET.format(netlumi), Key=AGENT_LIST_KEY.format(account), ExpectedBucketOwner=netlumi) except Exception as e: if error_code(e) in ('NoSuchKey', '404'): return None raise return json.loads(out['Body'].read()) def own_function_arn(context, partition, region, account): arn = getattr(context, 'invoked_function_arn', '') or '' if arn.count(':') >= 6: return ':'.join(arn.split(':')[:7]) return 'arn:%s:lambda:%s:%s:function:netlumi-onboarding-setup-%s' % (partition, region, account, account) def read_agent_tags(fn): try: return boto3.client('lambda').list_tags(Resource=fn).get('Tags') or {} except Exception as e: print('agent state tags: %s' % error_code(e)) return {} def write_agent_tags(fn, have, want): """Tags this function with the sync state, only when it changed (a tag write is a change event in the account).""" if all(have.get(k) == v for k, v in want.items()): return try: boto3.client('lambda').tag_resource(Resource=fn, Tags=want) except Exception as e: print('agent state tags: %s' % error_code(e)) def agent_trail(ct): out = ct.describe_trails(trailNameList=[AGENT_TRAIL], includeShadowTrails=False).get('trailList') or [] for t in out: if trail_name(t) == AGENT_TRAIL: return t return None def ensure_agent_trail(ct, bucket, partition, deadline): if agent_trail(ct): return while True: try: ct.create_trail(Name=AGENT_TRAIL, S3BucketName=bucket, IsMultiRegionTrail=True, IncludeGlobalServiceEvents=True, EnableLogFileValidation=True, TagsList=[{'Key': 'netlumi:managed-by', 'Value': 'netlumi'}]) return except Exception as e: code = error_code(e) if code == 'TrailAlreadyExistsException': return if code == 'InsufficientS3BucketPolicyException' and time.time() + TRAIL_RETRY_SECONDS < deadline: time.sleep(TRAIL_RETRY_SECONDS) continue raise def same_selectors(have, want): def norm(sels): return sorted(json.dumps({'Name': s.get('Name'), 'FieldSelectors': sorted( ({'Field': f.get('Field'), 'Equals': sorted(f.get('Equals') or []), 'StartsWith': sorted(f.get('StartsWith') or [])} for f in s.get('FieldSelectors') or []), key=lambda f: f['Field'])}, sort_keys=True) for s in sels or []) return norm(have) == norm(want) def delete_agent_trail(): try: boto3.client('cloudtrail').delete_trail(Name=AGENT_TRAIL) except Exception as e: if 'TrailNotFound' not in str(e) + error_code(e): print('agent trail cleanup: %s' % error_code(e)) def remove_agent_rules(account, regions, keep=0): """Deletes the -agents rules past the first keep in each region that exist (DescribeRule first: no failed call on every sync).""" for region in sorted(set(regions)): ev = boto3.client('events', region_name=region) remove_rules(ev, account, region, existing_rules(ev, account, AGENT_RULE_IDS[keep:])) def existing_rules(ev, account, rule_ids): out = [] for rid in rule_ids: try: ev.describe_rule(Name=RULE.format(account) + rule_suffix(rid)) out.append(rid) except Exception as e: if 'ResourceNotFound' not in error_code(e) + str(e): out.append(rid) return out def month_start(now): """(epoch seconds of the 1st of now's month, 'YYYY-MM'), UTC.""" t = time.gmtime(now) return calendar.timegm((t.tm_year, t.tm_mon, 1, 0, 0, 0)), '%04d-%02d' % (t.tm_year, t.tm_mon) def agent_month_usd(account, regions, rules, now): """Projected cost of this month's agent events (in the account): the -agents rules' MatchedEvents (AWS/Events, free) so far, scaled to the month.""" start, _ = month_start(now) matched = 0.0 for region in sorted(set(regions)): cw = boto3.client('cloudwatch', region_name=region) for rid in AGENT_RULE_IDS[:rules]: out = cw.get_metric_statistics(Namespace='AWS/Events', MetricName='MatchedEvents', Dimensions=[{'Name': 'RuleName', 'Value': RULE.format(account) + rule_suffix(rid)}], StartTime=int(start), EndTime=int(now), Period=86400, Statistics=['Sum']) matched += sum(p.get('Sum', 0) for p in out.get('Datapoints') or []) t = time.gmtime(now) days = calendar.monthrange(t.tm_year, t.tm_mon)[1] elapsed = max(now - start, 86400.0) return matched * (days * 86400.0 / elapsed) * AGENT_EVENT_USD def rules_tag(regions, rules): return ('%d %s' % (rules, ' '.join(sorted(regions)))).strip()[:256] def parse_rules_tag(value): parts = (value or '').split() if not parts or not parts[0].isdigit(): return [], 0 return [r for r in parts[1:] if AGENT_REGION.match(r)], min(int(parts[0]), len(AGENT_RULE_IDS)) def enforce_cap(ct, cfg, tags, now, regions, rules): """True when this month's agent logging is over the cap: the trail stops logging until the 1st of next month. Works without Netlumi.""" _, month = month_start(now) capped = tags.get(TAG_CAPPED) == month if not capped and regions and rules: try: capped = agent_month_usd(cfg['account'], regions, rules, now) > cfg['cap'] except Exception as e: print('agent cost: %s' % error_code(e)) try: logging = ct.get_trail_status(Name=AGENT_TRAIL).get('IsLogging') except Exception as e: if 'TrailNotFound' not in error_code(e) + str(e): raise return capped if capped and logging: ct.stop_logging(Name=AGENT_TRAIL) elif not capped and not logging: ct.start_logging(Name=AGENT_TRAIL) return capped def put_agent_rules(account, netlumi, region, patterns): """None when the region's -agents rules are in place, else the reason.""" ev = boto3.client('events', region_name=region) role = 'arn:aws:iam::%s:role/NetlumiEventBridgeForwarderRole-%s' % (account, account) target = {'Id': 'NetlumiEventBus', 'Arn': bus_for(region, netlumi), 'RoleArn': role} try: for rid, pat in zip(AGENT_RULE_IDS, patterns): name = RULE.format(account) + rule_suffix(rid) try: have = ev.describe_rule(Name=name) except Exception as e: if 'ResourceNotFound' not in error_code(e) + str(e): raise have = {} if have.get('EventPattern') == pat and have.get('State') == 'ENABLED_WITH_ALL_CLOUDTRAIL_MANAGEMENT_EVENTS': continue put_rule(ev, RULE.format(account) + rule_suffix(rid), dict(State='ENABLED_WITH_ALL_CLOUDTRAIL_MANAGEMENT_EVENTS', EventPattern=pat, Description='Netlumi: what the AI agents\' tool roles do (data events, model calls).'), target) except Exception as e: print('agent rules %s: %s' % (region, error_code(e))) return failure_of(region, 'agents', e)['code'] remove_rules(ev, account, region, existing_rules(ev, account, AGENT_RULE_IDS[len(patterns):])) return None def agent_trail_problem(e): """(state, warning) of a trail CloudTrail refused.""" code = error_code(e) low = (code + ' ' + str(e)).lower() if code == 'MaximumNumberOfTrailsExceededException': return 'limit', {'part': 'agent_monitoring', 'code': 'trail_limit', 'message': ('This region already has 5 trails (the AWS limit), so Netlumi could not create ' 'netlumi-agent-activity to log what your AI agents do. Remove a trail you no longer ' 'need; Netlumi retries every 15 minutes.')} if code in DENIED_CODES or 'accessdenied' in code.lower(): why = 'a service control policy' if 'service control policy' in low else ( 'a permissions boundary' if 'permissions boundary' in low else 'your organization\'s policy') return 'denied', {'part': 'agent_monitoring', 'code': 'trail_denied', 'message': ('%s denies creating or changing the netlumi-agent-activity trail (%s), so what your AI ' 'agents do is not logged. Ask your cloud team to allow it for this trail.' % (why, code))[:400]} return 'error', warning('agent_monitoring', e, 'AI agent monitoring trail') def agent_config(props, partition, region, account): regions = regions_of(props) try: cap = float(props.get('AgentMonitoringMonthlyCapUSD') or props.get('CapUSD') or 10) except (TypeError, ValueError): cap = 10.0 return {'enabled': props.get('AgentMonitoring') == 'enabled', 'cap': cap, 'regions': regions, 'realtime': props.get('RealtimeForwarding') != 'disabled', 'bucket': props.get('AgentActivityBucket') or '', 'netlumi': props.get('NetlumiAccountId') or '', 'partition': partition, 'region': region, 'account': account} def agent_teardown(cfg, fn, tags, state): delete_agent_trail() remove_agent_rules(cfg['account'], set(cfg['regions']) | {'us-east-1'}) write_agent_tags(fn, tags, {TAG_STATE: state, TAG_REASON: 'none'}) return state, [] def agent_sync(cfg, fn, deadline, now=None): """(state, warnings). state: off | waiting_for_agents | on | partly | limit | denied | capped | list_rejected | error (the callback's agentTrail).""" now = time.time() if now is None else now tags = read_agent_tags(fn) if not cfg['enabled'] or not cfg['realtime'] or not cfg['bucket']: # Disabled (or nothing can be forwarded): no trail, no rules, no cost. if tags.get(TAG_STATE) in (None, 'off'): return 'off', [] return agent_teardown(cfg, fn, tags, 'off') account, partition = cfg['account'], cfg['partition'] ct = boto3.client('cloudtrail') _, month = month_start(now) def keep_last(state, reason, warnings=()): # The last good list stays applied; the cap is still enforced. regions, rules = parse_rules_tag(tags.get(TAG_RULES)) want = {TAG_STATE: state, TAG_REASON: reason} try: if enforce_cap(ct, cfg, tags, now, regions, rules): want[TAG_CAPPED] = month except Exception as e: print('agent cap: %s' % error_code(e)) write_agent_tags(fn, tags, want) return state, list(warnings) try: applied = int(tags.get(TAG_VERSION) or 0) except ValueError: applied = 0 try: lst = read_agent_list(cfg['netlumi'], account) except Exception as e: print('agent list: %s' % error_code(e)) return keep_last('error', 'list_unreadable', [warning('agent_monitoring', e, 'AI agent monitoring could not read its list from Netlumi')]) if lst is None: write_agent_tags(fn, tags, {TAG_STATE: 'waiting_for_agents', TAG_REASON: 'none'}) return 'waiting_for_agents', [] problem = agent_list_problem(lst, account, partition, cfg['cap'], applied, agent_role_exists) if problem: print('agent list refused: %s' % problem) return keep_last('list_rejected', problem) ids = agent_monitored(lst) if not ids: # No agent left: the trail goes (its slot and its cost), as do the rules. delete_agent_trail() remove_agent_rules(account, set(cfg['regions']) | {'us-east-1'}) write_agent_tags(fn, tags, {TAG_STATE: 'waiting_for_agents', TAG_VERSION: str(lst['version']), TAG_REASON: 'none', TAG_RULES: '0'}) return 'waiting_for_agents', [] selectors, _ = agent_selectors(lst, account, partition) patterns = agent_rule_patterns(lst, account, partition) try: ensure_agent_trail(ct, cfg['bucket'], partition, deadline) have = ct.get_event_selectors(TrailName=AGENT_TRAIL) if have.get('EventSelectors') or not same_selectors(have.get('AdvancedEventSelectors'), selectors): ct.put_event_selectors(TrailName=AGENT_TRAIL, AdvancedEventSelectors=selectors) except Exception as e: state, w = agent_trail_problem(e) write_agent_tags(fn, tags, {TAG_STATE: state, TAG_REASON: w['code']}) return state, [w] # The -agents rules go in every forwarded region where an agent role has resources. want = sorted({r for i in ids for r in i.get('regions') or []}) forwarded = [r for r in want if r in cfg['regions']] reasons = [] if len(forwarded) < len(want): reasons.append('region_not_forwarded') if [r for r in forwarded if put_agent_rules(account, cfg['netlumi'], r, patterns)]: reasons.append('rule_failed') remove_agent_rules(account, [r for r in set(cfg['regions']) | {'us-east-1'} if r not in want]) if lst.get('dropped'): reasons.append('value_budget') try: capped = enforce_cap(ct, cfg, tags, now, forwarded, len(patterns)) except Exception as e: state, w = agent_trail_problem(e) write_agent_tags(fn, tags, {TAG_STATE: state, TAG_REASON: w['code']}) return state, [w] state = 'capped' if capped else ('partly' if reasons else 'on') write_agent_tags(fn, tags, {TAG_STATE: state, TAG_VERSION: str(lst['version']), TAG_REASON: (' '.join(reasons) or 'none')[:256], TAG_CAPPED: month if capped else 'none', TAG_RULES: rules_tag(forwarded, len(patterns))}) return state, [] def agent_limit_ahead(trails, region): """A warning at setup when the trail could not be created once an agent is found: this region already holds 5 trails and none is Netlumi's.""" here = [t for t in trails or [] if t.get('HomeRegion', region) == region] if len(here) >= 5 and not any(trail_name(t) == AGENT_TRAIL for t in here): return agent_trail_problem(_TrailQuota())[1] return None class _TrailQuota(Exception): response = {'Error': {'Code': 'MaximumNumberOfTrailsExceededException'}} def agent_sync_event(event, context): """The 15-minute schedule (netlumi-agent-sync-).""" account = event.get('Account') or '' partition, region = event.get('Partition') or 'aws', event.get('Region') or '' fn = own_function_arn(context, partition, region, account) if not account or not region: parts = fn.split(':') partition, region, account = parts[1], parts[3], parts[4] cfg = agent_config(event, partition, region, account) deadline = time.time() + remaining_seconds(context) - 30 state, warnings = agent_sync(cfg, fn, deadline) print('agent sync: %s%s' % (state, ''.join(' ' + w['code'] for w in warnings))) return {'state': state} # --- end of AI agent monitoring --- def watch(event, context): props, rtype = event['ResourceProperties'], event['RequestType'] stack_id, region = event['StackId'], event['StackId'].split(':')[3] if rtype == 'Create': notify(props, 'started', timeout=15, stackId=stack_id, region=region) return status = stack_status(stack_id) if rtype == 'Update' and status == 'UPDATE_ROLLBACK_IN_PROGRESS': notify(event['OldResourceProperties'], 'failed', timeout=15, stackId=stack_id, region=region, reason=first_failure(stack_id)) elif rtype == 'Update': notify(props, 'started', timeout=15, stackId=stack_id, region=region) elif rtype == 'Delete' and 'ROLLBACK' in status: notify(props, 'failed', timeout=15, stackId=stack_id, region=region, reason=first_failure(stack_id)) def replaced_in_update(stack_id): """True when CloudFormation deletes a superseded physical id in an update's cleanup (the trail marker changed): the current one owns the rules, DNS, EKS access and trail.""" try: status = stack_status(stack_id) except Exception as e: print('stack status: %s' % type(e).__name__) return False return status.startswith('UPDATE') and status.endswith('CLEANUP_IN_PROGRESS') def setup(event, context): props, rtype = event['ResourceProperties'], event['RequestType'] stack_id = event['StackId'] partition, region, account = stack_id.split(':')[1], stack_id.split(':')[3], stack_id.split(':')[4] base = 'netlumi-setup-' + account current = event.get('PhysicalResourceId') or '' old = event.get('OldResourceProperties') deadline = deadline_of(context) regions = regions_of(props) plan = rule_plan(props) if rtype == 'Delete': if current in (base, base + TRAIL_OWNED) and not replaced_in_update(stack_id): delete_rules(account, sorted(plan), deadline) if dns_vpcs(props): for r in regions: try: remove_dns(r, deadline=deadline) except Exception as e: print('dns cleanup %s: %s' % (r, type(e).__name__)) if eks_clusters(props): eks = boto3.client('eks', region_name=region) for c in eks_clusters(props): remove_eks(eks, c, props.get('ScannerRoleArn') or '') if current.endswith(TRAIL_OWNED): delete_trail() if props.get('AgentMonitoring'): # The agent trail and rules are always this stack's. delete_agent_trail() remove_agent_rules(account, set(regions) | {'us-east-1'}) notify(props, 'deleted', timeout=10, accountId=account) return respond(event, context, True, pid=current) states = region_states() if rtype == 'Update' and stack_status(stack_id) == 'UPDATE_ROLLBACK_IN_PROGRESS': # props: the previous properties; Old...: the failed ones. delete_rules(account, sorted(set(rule_plan(old)) - set(plan)), deadline) deployed, failed = put_rules(account, props, plan, deadline, states) warnings = sync_dns(account, props, old, deadline) + sync_eks(region, partition, props, old) keep = current if current in (base, base + TRAIL_OWNED) else base # Netlumi's record follows the rollback too (the # restored properties carry the version the stack is back on). try: report_update(props, account, stack_id, region, rolledBack=True, regionsDeployed=deployed, regionsFailed=failed, realtimeForwarding='disabled' if props.get('RealtimeForwarding') == 'disabled' else 'enabled') except Exception as e: print('update report: %s' % type(e).__name__) return respond(event, context, True, pid=keep, data=rules_data(deployed, failed, warnings)) try: if rtype == 'Update': delete_rules(account, sorted(set(rule_plan(old)) - set(plan)), deadline) # A region whose rule fails (opt-in region off, SCP) is reported, not fatal. deployed, failed = put_rules(account, props, plan, deadline, states) except Exception as e: reason = str(e)[:600] notify(props, 'failed', timeout=15, stackId=stack_id, region=region, reason=reason) return respond(event, context, False, reason=reason, pid=base) warnings = sync_dns(account, props, old if rtype == 'Update' else None, deadline) warnings += sync_eks(region, partition, props, old if rtype == 'Update' else None) trail, trail_name = 'off', '' owned = rtype == 'Update' and current.endswith(TRAIL_OWNED) trails = list_trails() if props.get('EnableCloudTrail') == 'true': try: trail, trail_name, trail_warnings = ensure_trail(props.get('CloudTrailBucket') or '', deadline, trails) warnings += trail_warnings owned = owned or trail in ('created', 'ours') except Exception as e: trail = 'failed' warnings.append(trail_warning(e)) # AI agent monitoring, applied now as on every sync. agent_state = '' if props.get('AgentMonitoring'): try: agent_state, agent_warnings = agent_sync(agent_config(props, partition, region, account), own_function_arn(context, partition, region, account), deadline) warnings += agent_warnings if agent_state == 'waiting_for_agents': ahead = agent_limit_ahead(trails, region) if ahead: warnings.append(ahead) except Exception as e: agent_state = 'error' warnings.append(warning('agent_monitoring', e, 'AI agent monitoring')) gov = governance(trails) org_id = '' if props.get('AccountType') == 'organization': try: org_id = boto3.client('organizations').describe_organization()['Organization']['Id'] except Exception as e: print('organization id: %s' % type(e).__name__) for w in warnings: print('warning %s: %s' % (w['part'], w['code'])) pid = base + (TRAIL_OWNED if owned else '') data = rules_data(deployed, failed, warnings) data['CloudTrail'] = trail migrate_eks = rtype == 'Update' and template_version(old) < 12 and bool(eks_clusters(props)) def report(): # What the completed and the updated callbacks both carry. return dict(regionsDeployed=deployed, regionsFailed=failed, warnings=warnings, realtimeForwarding='disabled' if props.get('RealtimeForwarding') == 'disabled' else 'enabled', # Which trail feeds real time, the regions the # account has not enabled, Control Tower. trail=trail, trailName=trail_name, governance=gov, regionsNotEnabled=not_enabled(regions + list(plan), states), # AI agent monitoring. agentMonitoring=props.get('AgentMonitoring', ''), agentTrail=agent_state, agentMonitoringCapUsd=int(agent_config(props, partition, region, account)['cap']), # Every other choice made in the stack. stackOptions=stack_options(props)) if rtype == 'Update' and (old or {}).get('RegistrationId') == props['RegistrationId']: # Same link: already connected, the one-time secret spent. The update # is reported anyway, so Netlumi's record follows the stack # however it was updated (console, CLI, StackSets, a pipeline). try: report_update(props, account, stack_id, region, **report()) except Exception as e: print('update report: %s' % type(e).__name__) respond(event, context, True, pid=pid, data=data) else: ok, body = notify(props, 'completed', accountId=account, roleArn=props['ScannerRoleArn'], orgBootstrapRoleArn=props.get('OrgBootstrapRoleArn', ''), organizationId=org_id, # The registration the stack carried before this update. previousRegistrationId=(old or {}).get('RegistrationId', '') if rtype == 'Update' else '', stackId=stack_id, region=region, **report()) if not ok: return respond(event, context, False, pid=pid, reason='Netlumi: ' + str(body.get('message', 'registration failed'))) respond(event, context, True, pid=pid, data=data) if migrate_eks: resync_eks_after_cleanup(stack_id, region, partition, props, time.time() + remaining_seconds(context) - WATCHDOG_SECONDS) def report_update(props, account, stack_id, region, **fields): """Tells Netlumi what an update of an already connected stack left in place (the 'updated' callback, authenticated by the stack's callback secret). Never fails the update: the stack is already connected, and an unrecorded report is logged here and leaves Netlumi's record as it was.""" ok, body = notify(props, 'updated', timeout=30, accountId=account, stackId=stack_id, region=region, **fields) if ok: print('update reported: template v%s' % (props.get('TemplateVersion') or 'unknown')) else: print('update not recorded by Netlumi: %s' % str(body.get('code') or body.get('message') or 'unknown')[:200]) return ok def stack_options(props): """The stack's choices beyond those the callback reports elsewhere. Lists are counted, never sent; the boundary is only 'set' or 'none'. A choice the trigger did not pass is '' (unknown to Netlumi, never off).""" def pick(name, values): return values.get(props.get(name), '') if name in props else '' def flag(name, on='true'): return pick(name, {on: 'enabled', 'false' if on == 'true' else 'disabled': 'disabled'}) def count(name): v = props.get(name) or [] if isinstance(v, str): v = v.split(',') return len({x.strip() for x in v if x.strip()}) return {'createCloudTrail': flag('EnableCloudTrail'), 'forwardAIInvocations': flag('ForwardAIInvocationEvents'), 'dnsQueryLogging': flag('EnableDnsQueryLogging'), 'dnsQueryLoggingVpcCount': len(dns_vpcs(props)), 'eksClusterCount': len(eks_clusters(props)), 'stateLockDiscovery': flag('StateLockDiscovery', 'enabled'), 'stateBucketCount': count('StateBuckets'), 'extraStateObjectCount': count('ExtraStateObjectArns'), 'permissionsBoundary': pick('PermissionsBoundary', {'set': 'set', 'none': 'none'}), 'orgBootstrapAccess': pick('OrgBootstrapAccess', {'enabled': 'enabled', 'removed': 'removed'})} def start_watchdog(event, context): """Answers CloudFormation before the Lambda times out if nothing else has: a create or update fails with a reason, a delete succeeds.""" secs = remaining_seconds(context) - WATCHDOG_SECONDS if secs <= 0: return None account = event['StackId'].split(':')[4] def fire(): print('setup ran out of time') respond(event, context, event['RequestType'] == 'Delete', pid=event.get('PhysicalResourceId') or ('netlumi-setup-' + account), reason='Netlumi setup ran out of time before it finished (see CloudWatch Logs). Retry with a fresh link from Netlumi.') t = threading.Timer(secs, fire) t.daemon = True t.start() return t def handler(event, context): print('%s %s' % (event.get('RequestType'), event.get('LogicalResourceId'))) timer = None if event.get('Mode') == 'agent-sync': # The 15-minute schedule, not a CloudFormation request. return agent_sync_event(event, context) try: if event['ResourceProperties'].get('Mode') == 'watch': try: watch(event, context) except Exception as e: print('watch: %s' % type(e).__name__) return respond(event, context, True, pid='netlumi-stack-watch', data={'BootstrapNotAfter': bootstrap_not_after()}) timer = start_watchdog(event, context) setup(event, context) except Exception as e: print('unhandled: %s' % type(e).__name__) respond(event, context, event['RequestType'] == 'Delete', reason=str(e)[:600]) finally: if timer: timer.cancel() NetlumiSetupTrigger: Type: Custom::NetlumiSetup # The scanner role is a dependency through ScannerRoleArn. Deleted before # the log group its function writes to. DependsOn: - NetlumiEventBridgeForwarderRole - NetlumiSetupLogGroup Properties: ServiceToken: !GetAtt NetlumiSetupFunction.Arn ServiceTimeout: 900 Mode: setup RegistrationId: !Ref RegistrationId CallbackSecret: !Ref CallbackSecret CallbackUrl: !Ref CallbackUrl NetlumiAccountId: !Ref NetlumiAccountId AccountType: !Ref AccountType Regions: !Ref Regions EnableCloudTrail: !Ref EnableCloudTrail ForwardAIInvocationEvents: !Ref ForwardAIInvocationEvents RealtimeForwarding: !Ref RealtimeForwarding GlobalEventsRegion: us-east-1 SignInEventsRegions: us-east-2,us-west-2,eu-north-1,ap-southeast-2 EnableDnsQueryLogging: !Ref EnableDnsQueryLogging DnsQueryLoggingVpcIds: !Ref DnsQueryLoggingVpcIds # Makes the trigger wait for the DNS policy when it exists. DnsQueryLoggingPolicy: !If [CreateDnsQueryLogging, !Ref NetlumiDnsQueryLoggingPolicy, ''] CloudTrailMode: !If [CreateCloudTrailBucket, created, 'off'] CloudTrailBucket: !If [CreateCloudTrailBucket, !Ref NetlumiCloudTrailBucket, ''] # Makes the trigger wait for the bucket policy: CloudTrail refuses a # trail whose bucket does not let it write. CloudTrailBucketPolicy: !If [CreateCloudTrailBucket, !Ref NetlumiCloudTrailBucketPolicy, ''] EksClusters: !Ref EksClusters # Makes the trigger wait for the EKS policy when it exists. EksAccessPolicy: !If [HasEksClusters, !Ref NetlumiEksAccessPolicy, ''] # The remaining choices, for the completed callback only. # No ARN or name leaves the account: the callback reports counts and # whether a boundary is set. OrgBootstrapAccess: !Ref OrgBootstrapAccess StateBuckets: !Ref StateBuckets ExtraStateObjectArns: !Ref ExtraStateObjectArns StateLockDiscovery: !Ref StateLockDiscovery PermissionsBoundary: !If [HasPermissionsBoundary, set, none] AgentMonitoring: !Ref AgentMonitoring AgentMonitoringMonthlyCapUSD: !Ref AgentMonitoringMonthlyCapUSD AgentActivityBucket: !If [AgentMonitoringEnabled, !Ref NetlumiAgentActivityBucket, ''] # Makes the trigger wait for the bucket policy and the sync's grants, and # outlive them on deletion (the trail and rules go first). AgentActivityBucketPolicy: !If [AgentMonitoringEnabled, !Ref NetlumiAgentActivityBucketPolicy, ''] AgentMonitoringPolicy: !If [AgentMonitoringEnabled, !Ref NetlumiAgentMonitoringPolicy, ''] ScannerRoleArn: !GetAtt NetlumiScannerRole.Arn OrgBootstrapRoleArn: !If - CreateOrgBootstrapRole - !GetAtt NetlumiOrgBootstrapRole.Arn - '' TemplateVersion: '23' # The agent sync every 15 minutes (after the setup, so the two # never race on a new stack). NetlumiAgentSyncSchedule: Type: AWS::Events::Rule Condition: AgentMonitoringEnabled DependsOn: NetlumiSetupTrigger Properties: Name: !Sub 'netlumi-agent-sync-${AWS::AccountId}' Description: Netlumi AI agent monitoring sync (applies the agent role list Netlumi publishes for this account) ScheduleExpression: rate(15 minutes) State: ENABLED Targets: - Id: NetlumiAgentSync Arn: !GetAtt NetlumiSetupFunction.Arn Input: !Sub - '{"Mode":"agent-sync","TemplateVersion":"23","AgentMonitoring":"${AgentMonitoring}","AgentMonitoringMonthlyCapUSD":"${AgentMonitoringMonthlyCapUSD}","Regions":"${RegionList}","RealtimeForwarding":"${RealtimeForwarding}","NetlumiAccountId":"${NetlumiAccountId}","AgentActivityBucket":"${NetlumiAgentActivityBucket}","Account":"${AWS::AccountId}","Region":"${AWS::Region}","Partition":"${AWS::Partition}"}' - RegionList: !Join [',', !Ref Regions] NetlumiAgentSyncPermission: Type: AWS::Lambda::Permission Condition: AgentMonitoringEnabled Properties: FunctionName: !Ref NetlumiSetupFunction Action: 'lambda:InvokeFunction' Principal: events.amazonaws.com SourceAccount: !Ref 'AWS::AccountId' SourceArn: !GetAtt NetlumiAgentSyncSchedule.Arn Outputs: ScannerRoleArn: Description: ARN of the Netlumi scanner IAM role Value: !GetAtt NetlumiScannerRole.Arn DataReadGuardPolicyArns: Description: The scanner role's data-read guard policies (explicit Deny) Value: !Join [',', [!Ref NetlumiDataReadGuard, !Ref NetlumiDataReadGuardScoped]] OrgBootstrapRoleArn: Condition: CreateOrgBootstrapRole Description: The organization bootstrap role (usable until OrgBootstrapNotAfter) Value: !GetAtt NetlumiOrgBootstrapRole.Arn OrgBootstrapNotAfter: Condition: CreateOrgBootstrapRole Description: Netlumi cannot assume the bootstrap role after this time (UTC) Value: !GetAtt NetlumiStackWatch.BootstrapNotAfter EventBridgeForwarderRoleArn: Description: ARN of the EventBridge cross-account forwarder role Value: !GetAtt NetlumiEventBridgeForwarderRole.Arn RealTimeForwardingWarnings: Description: Regions where real-time forwarding is off (rule or delivery blocked), or none Value: !GetAtt NetlumiSetupTrigger.RegionsFailed SetupWarnings: Description: Optional parts that could not be set up (CloudTrail trail, DNS query logging, EKS access), or none Value: !GetAtt NetlumiSetupTrigger.Warnings