Home

Awesome

AWS Incident Response

Investigation of API activity using Athena and notification of actions using EventBridge

Use

For a slighly more pleasant viewing experience, use the GitHub pages link: https://easttimor.github.io/aws-incident-response/

For those on GitHub Pages already, the repo additionally contains Terraform resources for deploying Event Rules to catch high risk API events.

Introduction

This project explores useful CloudTrail events that support incident response and detection of misconfigurations. Documenting the queries and filters used to identify these CloudTrail events helps to:

Mis-configurations are important events to identify early. These configurations may introduce a vulnerability, but may also be an indicator of compromise.

Whether executed manually or by automating, this information may be used to develop incident response playbooks. These types of formalization activities promote a consistent, efficient, and effective response to security incidents.

NEW 2020-08-04 Addition of VPC Flow Log queries via Athena

Table of Contents

Why Athena?

CloudTrail logs should be stored and archived in S3, where they are essentially useless unless integrated with another product or service. Amazon Athena allows you to query these JSON-formatted logs using standard SQL. This approach gives access to a potentially massive amount of CloudTrail data without the cost and effort of implementing Splunk, ElasticSearch, or storing in another database.

Getting Started

Athena charges for the data scanned for each query. You may only return one record, but will be charged for all data queried to match that record. By default, S3 data is not indexed, so Athena will inefficiently scan a LOT of data. There are also S3 charges (GET requests) that factor in at larger scale.

https://aws.amazon.com/athena/pricing/

But, we can make Athena faster and cost effective by creating partitions. Implement CloudTrail Partitioner. By default, Partitioner will create a virtual table for each AWS account so you don't scan an entire aggregated bucket. Partitioner will also add the following partitions:

Including these partitions in your SQL statements, as shown throughout this project, significantly improves query performance by limiting the amount of data scanned. Limiting the amount of data scanned saves money. For ad-hoc queries, such as those used for incident response, this cost is negligible.

https://github.com/duo-labs/cloudtrail-partitioner

Alternatives

Why EventBridge?

EventBridge builds on CloudWatch Events and uses the same APIs. This is essentially CloudWatch Events.

Querying CloudTrail, even if automated, is best suited for ad-hoc response to finding misconfigurations and investigating incidents. Many of these configuration-related indicators of compromise can be detected in near real time. EventBridge allows for these pre-defined CloudTrail events to be filtered and integrated with numerous alerting methods (e.g. SNS) and event flows (e.g. Lambda, 3rd party SIEM).

This approach is independent of the Athena queries, but both approaches complement each other.

The Terraform section of this project repo includes deployable event filters with a basic SNS notification. Where applicable, Athena queries on this page are reflected in these event filters.

EventBridge is cheap for this use case.

https://aws.amazon.com/eventbridge/pricing/

Incidents

The following section builds a collection of common incidents and the Athena queries that may prove useful in response. These queries attempt to explain timeline, scope, impact, and surface indicators of compromise.

General IAM Investigation

Some of the queries in this section were inspired by the following Reference: https://wellarchitectedlabs.com/security/300_labs/300_incident_response_with_aws_console_and_cli/2_iam/

API Errors

select eventTime, eventSource, eventName, errorCode, errorMessage, responseElements, awsRegion, userIdentity.arn, sourceIPAddress, userAgent
from cloudtrail_000000000000
and year = '####' and month = '##'
and errorCode in ('Client.UnauthorizedOperation','Client.InvalidPermission.NotFound','Client.OperationNotPermitted','AccessDenied')
order by eventTime desc
limit 25

Activity from potentially malicious source ip

select eventTime, eventSource, eventName, awsRegion, userIdentity.arn, sourceIPAddress, userAgent
from cloudtrail_000000000000
and year = '####' and month = '##'
and sourceIPAddress = 'x.x.x.x'
order by eventTime desc
limit 25

S3 ListBuckets

The ListBuckets API action, used to enumerate buckets within an AWS account, is a potential indicator of compromise.

select eventTime, eventSource, eventName, awsRegion, errorCode, errorMessage, userIdentity.arn, sourceIPAddress, userAgent
from cloudtrail_000000000000
and year = '####' and month = '##'
and eventName = 'ListBuckets'
limit 25

Incident: Access Key Exposure

We are looking for the following:

These queries must include all AWS Regions.

Find the IAM Principal and AWS Account associated with Access Key

select useridentity.principalId, useridentity.arn, useridentity.accountId, region
from cloudtrail_000000000000
where year = '####' and month = '##'
and useridentity.accesskeyid = 'AKIAxxxxxxxxxxxxxxxx'
limit 1

All key usage

Looking for high risk events that indicate establishing persistence, escalating permissions, resource creation (denial-of-wallet), data access (exfiltration). These actions may include:

Pay attention to all actions identified for priviledge escalation in this document.

select eventtime, eventsource, eventname, sourceipaddress, errorcode, useragent
from cloudtrail_000000000000
where useridentity.accesskeyid = 'AKIAxxxxxxxxxxxxxxxx'
and year = '####' and month = '##'

Activity for a specific IAM User

userIdentity.arn determined from the above query.

select eventtime, eventsource, eventname, errorcode, sourceipaddress, useragent
from cloudtrail_000000000000
where year = '####' and month = '##'
and userIdentity.arn = 'arn:aws:iam::000000000000:user/xxxxxxxx'
order by eventtime desc

Limit some noise.

select eventtime, eventsource, eventname, errorcode, sourceipaddress, useragent
from cloudtrail_000000000000
where year = '####' and month = '##'
and userIdentity.arn = 'arn:aws:iam::000000000000:user/xxxxxxxx'
and eventName not like 'Describe%'
and eventName not like 'List%'
and eventName not like 'Get%'
order by eventtime desc

Look for user agent anomalies for key

Determine "normal" for user agent string

select useragent, count(*) as total
from cloudtrail_000000000000
where useridentity.accesskeyid = 'AKIAxxxxxxxxxxxxxxxx'
and year = '####' and month = '##'
group by useragent
order by total desc

Assess activity for a specific user agent string; Conditions may be added to limit to a specific user or credential.

select eventtime, eventsource, eventname, sourceipaddress, errorcode
from cloudtrail_000000000000
where year = '####' and month = '##'
and useragent = 'seeAbove'

Group by user to include all access for a single user. This approach would be helpful if keys are rotated or a console login is used.

This value format is for an IAM user and not an assumed role

select useragent, count(*) as total
from cloudtrail_000000000000
where year = '####' and month = '##'
and userIdentity.userName = 'xxxxxx'
group by useragent
order by total desc

Look for source ip anomalies

Group by sourceIpAddress for a specific access key.

select sourceipaddress, count(*) as total
from cloudtrail_000000000000
where year = '####' and month = '##'
and useridentity.accesskeyid = 'AKIAxxxxxxxxxxxxxxxx'
group by sourceipaddress
order by total desc

Assess all activity for a specific source IP, typically one believed to be an adversary in the case of a compromised key.

select eventtime, eventsource, eventname, errorcode, useragent
from cloudtrail_000000000000
where sourceipaddress = 'seeAbove'
and year = '####'
and month = '##'

Group by user to include multiple credentials for a single user. This approach would be helpful if keys are rotated or a console login is used.

This value format is for an IAM user and not an assumed role

select sourceipaddress, count(*) as total
from cloudtrail_000000000000
where year = '####' and month = '##'
and userIdentity.userName = 'xxxxxx'
group by sourceipaddress
order by total desc

Incident: EC2 Instance Compromise

EC2 instances may have an IAM Role attached to them. The combination of the instance and the role is called an "instance profile". When the role is assumed, the EC2 instance ID is used as the session name part of the Principal ARN in CloudTrail. We can identify actions of EC2 instances using the clause useridentity.principalid like '%:i-%' or a specific EC2 instance useridentity.principalid like '%:i-00000000000000000'

The actions of EC2 instances will typically be repetitive and persistent, because all actions are presumed to be initiated by software and not a human. Play close attention to any anomalous API calls. An attacker with access to an EC2 instance has access to any IAM permissions granted to that instance via the instance profile.

Recent API calls with a specific instanceId as the target resource

Very inefficient I know - room for improvement here

The CloudTrail UI provides resource name as search criteria. Note that resource name is not an actual key in the JSON so they're abstracting some query magic. Relevant events may not be included in this CloudTrail API query - otherwise stated, the below Athena query will show you more events for better or worse.

select eventTime, eventName, eventSource, userIdentity.arn
from cloudtrail_000000000000
where year = 'xxxx' and month = 'xx' and day = 'xx'
and (requestParameters like '%i-xxxxxxxxxxxxxxxxx%' or responseElements like '%i-xxxxxxxxxxxxxxxxx%')
and eventname not like 'Describe%'
and eventsource = 'ec2.amazonaws.com'
limit 25

EC2 Instance profile - most frequent API calls

select eventname, count(*) as total
from cloudtrail_000000000000
where year = '####' and month = '##' and day = '##'
and useridentity.principalid like '%:i-xxxxxxxxxxxxxxxxx'
group by eventname
order by total desc
limit 25

EC2 Instance profile - most denied API calls

select eventname, count(*) as total
from cloudtrail_000000000000
where year = '####' and month = '##' and day = '##'
and useridentity.principalid like '%:i-xxxxxxxxxxxxxxxxx'
and errorcode = 'AccessDenied'
group by eventname
order by total desc
limit 25

All EC2 Instance Profiles - most denied instance profiles

select useridentity.principalid, count(*) as total
from cloudtrail_000000000000
where year = '####' and month = '##' and day = '##'
and useridentity.principalid like '%:i-%'
and errorcode = 'AccessDenied'
group by useridentity.principalid
order by total desc
limit 25

All EC2 Instance Profiles - most denied event names

select eventsource,eventname,count(*) as total
from cloudtrail_000000000000
where year = '####' and month = '##' and day = '##'
and useridentity.principalid like '%:i-%'
and eventname <> 'AssumeRole'
and errorcode = 'AccessDenied'
group by eventsource,eventname
order by total desc
limit 25

EC2 Instance Profile interacting with IAM

EC2 instances rarely have a need for IAM actions. Include an and eventname <> clause if legitimate actions are found that muddy the search results.

select useridentity.principalid,eventsource,eventname,count(*) as total
from cloudtrail_000000000000
where year = '####' and month = '##' and day = '##'
and useridentity.principalid like '%:i-%'
and eventsource = 'iam.amazonaws.com'
group by useridentity.principalid,eventsource,eventname
order by total desc
limit 25

EC2 Instance Profile enumerating S3

EC2 instances should know exactly which S3 buckets to they need. The ListBuckets action is a strong indicator of compromise...or bad development practices.

select useridentity.principalid,eventsource,eventname,count(*) as total
from cloudtrail_000000000000
where year = '####' and month = '##' and day = '##'
and useridentity.principalid like '%:i-%'
and eventsource = 's3.amazonaws.com'
and eventname = 'ListBuckets'
group by useridentity.principalid,eventsource,eventname
order by total desc
limit 25

General Purpose Queries

These queries are useful for exploring potential issues and building upon for threat hunting.

Most common API actions for a given day

select eventname,count(*) as total
from cloudtrail_000000000000 
where year = '####' and month = '##' and day = '##'
group by eventname
order by total desc

Most common error codes

select errorcode, count(errorcode) as total
from cloudtrail_000000000000
where year = '####' and month = '##' and day = '##'
group by errorcode
order by total desc

Principals getting denied the most

select useridentity.principalid, count(*) as deniedactions
from cloudtrail_000000000000
where year = '####' and month = '##' and day = '##'
and errorcode = 'AccessDenied'
group by useridentity.principalid
order by deniedactions desc
limit 25

Common denied actions from specific principal (see above)

select eventname, count(*) as total
from cloudtrail_000000000000
where year = '####' and month = '##' and day = '##'
and errorcode = 'AccessDenied'
and useridentity.principalid = 'AROAxxxxxxxxxxxxxxxxx:i-xxxxxxxxxxxxxxxxx'
group by eventname
order by total desc

API Watchlist

these are better suited for event driven alerting, and that's what the Terraform in this repo aims to provide.

IAM

Root Credential Use

All Root account events

select * 
from cloudtrail_000000000000
where year = '####' and month = '##' 
and useridentity.type = 'Root'

Only ConsoleLogin event from Root

select *
from cloudtrail_000000000000
where year = '####' and month = '##' 
and eventname = 'ConsoleLogin'
and useridentity.type = 'Root'

Remove MFA from an IAM User

While IAM Policy may still include a condiion that requires MFA, removing MFA from an IAM User enables a pivot to that principal.

select *
from cloudtrail_000000000000
where year = '####' and month = '##' 
and eventname IN ('DeactivateMFADevice', 'DeleteVirtualMFADevice')

All IAM Changes

The IAM API has numerous actions for establishing persistence and expanding permissions. The following query removes read only actions.

select useridentity.principalid, eventname, count(*) as total
from cloudtrail_000000000000
where year = '####' and month = '##' and day = '##'
and eventsource = 'iam.amazonaws.com'
and eventname not like 'Get%' 
and eventname not like 'List%'
and eventname not like 'Generate%'
group by useridentity.principalid, eventname
order by total desc

Creation of IAM Principals

select useridentity.principalid, eventname, count(*) as total
from cloudtrail_000000000000
where year = '####' and month = '##' and day = '##'
and eventsource = 'iam.amazonaws.com'
and evenname in ('CreateUser','CreateRole','CreateServiceLinkedRole')

Privilege Escalation: IAM Policy

IAM Policy updates used to expand permissions of associated principals (IAM Users, IAM Roles).

select *
from cloudtrail_000000000000
where year = '####' and month = '##' and day = '##'
and eventName IN ('CreatePolicyVersion','SetDefaultPolicyVersion')
order by eventtime desc

Add/Update Credentials

select *
from cloudtrail_000000000000
where year = '####' and month = '##' and day = '##'
and eventName IN ('CreateAccessKey', 'UpdateAccessKey',
'CreateLoginProfile','UpdateLoginProfile',
'CreateVirtualMFADevice','DeactivateMFADevice','DeleteVirtualMFADevice','EnableMFADevice'
'CreateServiceSpecificCredential','UpdateServiceSpecificCredential',
'ResetServiceSpecificSredential','DeleteServiceSpecificCredential',
'UploadServerCertificate','DeleteServerCertificate',
'UploadSigningCertificate','UpdateSigningCertificate','DeleteSigningCertificate',
'UploadSSHPublicKey','UpdateSSHPublicKey','DeleteSSHPublicKey'
)
order by eventtime desc

Privilege Escalation: Adding permissions

Permission expansion may include disassociating a principal from an IAM Policy due to the removal of explicit Deny effects.

select *
from cloudtrail_000000000000
where year = '####' and month = '##' and day = '##'
and eventSource = 'iam.amazonaws.com' 
and eventName IN ('AttachUserPolicy', 'DetachUserPolicy',
'AttachRolePolicy', 'DetachRolePolicy',
'PutUserPolicy','PutGroupPolicy','PutRolePolicy',
'DeleteUserPolicy','DeleteGroupPolicy','DeleteRolePolicy',
'DeleteRolePermissionsBoundary')
order by eventtime desc

Privilege Escalation: Expand Access to an IAM Role

Updating a trust policy (UpdateAssumerolePolicy) allows a service, local account principal, or external account to assume the associate IAM Role and its permissions.

select *
from cloudtrail_000000000000
where year = '####' and month = '##' and day = '##'
and eventSource = 'iam.amazonaws.com' 
and eventName IN ('UpdateAssumeRolePolicy')
order by eventtime desc
ActionImpact
UpdateAssumeRolePolicyPersistence / Privilege escalation allowing an IAM User to assume an IAM Role and its associated permissions

Modify Federated Access

Adding a new identity provider (CreateSAMLProvider) also requires adding or updating an IAM's Role's trust policy.

Updating the metadata (UpdateSAMLProvider) for a SAML IdP would hijack an existing trusted relationship, but break existing federated access.

select *
from cloudtrail_000000000000
where year = '####' and month = '##' and day = '##'
and eventName IN ('CreateSAMLProvider','UpdateSAMLProvider','DeleteSAMLProvider',
'CreateOpenIDConnectProvider','DeleteOpenIDConnectProvider','UpdateOpenIDConnectProviderThumbprint',
'AddClientIDToOpenIDConnectProvider','RemoveClientIDFromOpenIDConnectProvider')
order by eventtime desc

S3

select *
from cloudtrail_000000000000
where year = '####' and month = '##' and day = '##'
and eventname in ('DeleteBucket','DeleteBucketPolicy',
'PutBucketAcl','PutBucketCORS','PutBucketPolicy','PutReplicationConfiguration',
'PutBucketLogging','PutEncryptionConfiguration','PutLifecycleConfiguration','PutObjectAcl',
'RestoreObject')
order by eventtime desc

For object access logging suppression, in addition to s3:PutBucketLogging see also cloudtrail:PutEventSelectors for logging configuration of data events.

ActionImpact
DeleteBucketStealth, if bucket contains logs
DeleteBucketPolicyExpand access if explicit denies are removed
PutBucketAclExpand access (exfil, pivot)
PutBucketCORSPotential for data exfiltration
PutBucketPolicyExpand access to bucket
PutReplicationConfigurationExpand access if target bucket is less restrictive, exfil data
PutBucketLoggingStealth, disable logging
PutEncryptionConfigurationDisable encryption, exfil cleartext data
PutLifecycleConfigurationExfil data if lifecycle rule incluces a more permissive target
PutObjectAclExpand access to object
RestoreObjectAccess an archived object

EC2

General

ActionImpact
GetPasswordDataCredential access
ModifyImageAttributeExfiltration
ModifySnapshotAttributeExfiltration

EBS Encryption Account-wide Setting

select *
from cloudtrail_000000000000
where year = '####' and month = '##' and day = '##'
and eventsource = 'ec2.amazonaws.com'
and eventname IN ('EnableEbsEncryptionByDefault','DisableEbsEncryptionByDefault')

Share EBS Snapshot

--user-ids 000000000000 is used to share with an external account

--group-names all is used to share publicly

select *
from cloudtrail_000000000000
where year = '####' and month = '##' and day = '##'
and eventsource = 'ec2.amazonaws.com'
and eventname = 'ModifySnapshotAttribute'

Modify UserData

An EC2 instance will execute UserData with root-level permissions on start/re-start. The instance must be in a stopped state to configure the userdata update.

select *
from cloudtrail_000000000000
where year = '####' and month = '##' and day = '##'
and eventname = 'ModifyInstanceAttribute'
and requestParameters like '%userData%'

Network Access

Network Access Control List (NACL) configuration

select *
from cloudtrail_000000000000
where year = '####' and month = '##' and day = '##'
and eventname IN (
   'CreateNetworkAcl',
   'CreateNetworkAclEntry',
   'DeleteNetworkAcl',
   'DeleteNetworkAclEncry'
   )
order by eventtime desc

Security group configuration

select *
from cloudtrail_000000000000
where year = '####' and month = '##' and day = '##'
and eventname IN (
   'AuthorizeSecurityGroupIngress',
   'AuthorizeSecurityGroupEgress'
   'CreateSecurityGroup',
   'ModifyInstanceAttribute'
   )
order by eventtime desc
ActionImpact
AuthorizeSecurityGroupIngressexpand EC2 isntance inbound traffic permissions (persistance, exfil, or exploit)
AuthorizeSecurityGroupEgressexpand EC2 instance initiated outbound traffic permissions (exfil data or pivot)
CreateSecurityGroupsupports ingress/egress permissions for any associated EC2 instance
ModifyInstanceAttributein this context, may be used to attach a security group to an EC2 instance network interface

Traffic Mirroring

Traffic Mirroring is a full packet capture (pcap) capability that may be used by an adversary to exfil secrets and sensitive data from unencrypted internal traffic. Within a VPC a traffic mirroring session is established with collection filter rules that identify what traffic to collect and forward to a target.

https://docs.aws.amazon.com/vpc/latest/mirroring/what-is-traffic-mirroring.html https://docs.aws.amazon.com/vpc/latest/mirroring/traffic-mirroring-filters.html

select *
from cloudtrail_000000000000
where year = '####' and month = '##' and day = '##'
and eventname IN (
    'CreateTrafficMirrorFilter',
    'CreateTrafficMirrorFilterRule',
    'CreateTrafficMirrorSession',
    'CreateTrafficMirrorTarget'
    )
ActionImpact
CreateTrafficMirrorFilterexpand EC2 isntance inbound traffic permissions (persistance, exfil, or exploit)
CreateTrafficMirrorFilterRuleconfigures the traffic to capture rules applied to a filter
CreateTrafficMirrorSessionestablishes a packet capture session
CreateTrafficMirrorTargetforwards captures traffic to an adversary controlled resource

Network Routing

Actions in this category have a high degree of legitimate use and are most helpful when correlated with other IoCs

select *
from cloudtrail_000000000000
where year = '####' and month = '##' and day = '##'
and eventname IN (
    'CreateRoute',
    'CreateRouteTable',
    'DeleteRouteTable',
    'DeleteRoute',
    'DisassociateRouteTable',
    'ReplaceRoute',
    'ReplaceRouteTableAssociation' 
    )
order by eventtime desc
ActionImpact
CreateRouteadds a new route (reroute/hijack traffic)
CreateRouteTableadds a new route table (reroute/hijack traffic)
DeleteRouteTableremove existing routing
DeleteRouteremove existing routing
DisassociateRouteTableremove existing routing
ReplaceRoutere-route existing traffic flow (hijack)
ReplaceRouteTableAssociationre-route traffic (reroute/hijack traffic)

The following events are less common than routing changes above, but should be correlated with other IoCs

These events are high impact to a VPC's ingress/egress traffic flow

select *
from cloudtrail_000000000000
where year = '####' and month = '##' and day = '##'
and eventname IN (
    'CreateCustomerGateway',
    'DeleteCustomerGateway',
    'AttachInternetGateway',
    'CreateInternetGateway',
    'DeleteInternetGateway',
    'DetachInternetGateway'
)
order by eventtime desc

Lambda

No API event in isolation should be concerning, so the importance of establishing context around Lambda service events is critical. Attackers may use existing Lambda functions to inherit elevated permissions as a pivot, for establishing persistence, or for accessing data. The actions indicated in this section are the most important to look for, but are by no means the only Lambda actions to care about in incident response.

ActionImpact
AddLayerVersionPermissionincrease permissions for lambda:InvokeFunction (e.g. external account)
AddPermissionincrease permissions for lambda:InvokeFunction (e.g. external account)
PublishLayerVersionrogue update of function code
PublishVersionrogue update of function code
UpdateFunctionCoderogue update of function code

Disruption and Evasion

CloudTrail

CloudTrail Disruption

select *
from cloudtrail_000000000000
where year = '####' and month = '##' and day = '##'
and eventname IN ('DeleteTrail','StopLogging','UpdateTrail',
'PutEventSelectors')
ActionImpact
DeleteTraildisrupt recording
StopLoggingdisrupt recording and delivery
UpdateTraildisrupt log delivery, minify * --no-include-global-service-events * --no-is-multi-region-trail * --no-enable-log-file-validation
PutEventSelectorsdisrupt data events for S3 and/or Lambda

To-do:

Config

Disrupt Config Recording, Evaluation, and Remediation

CloudTrail may still log resource configuration actions Goals here are to prevent resource configuration history (for forensics), non-compliance detection, and remediation Offensive capability may be introduced through Systems Manager Automation as remediation action

select *
from cloudtrail_000000000000
where year = '####' and month = '##' and day = '##'
and eventsource = 'config.amazonaws.com'
and eventname IN ('DeleteConfigRule','DeleteOrganizationConfigRule',
'DeleteConfigurationAggregator','DeleteConfigurationRecorder',
'DeleteConformancePack','DeleteOrganizationConformancePack',
'DeleteDeliveryChannel','PutDeliveryChannel',
'DeleteRemediationConfiguration','DeleteRetentionConfiguration',
'PutConfigRule', 'PutConfigurationAggregator','PutConformancePack',
'PutOrganizationConfigRule','PutOrganizationConformancePack',
'PutRemediationConfigurations','PutRemediationExceptions',
'PutRetentionConfiguration',
'StopConfigurationRecorder')
ActionImpact
DeleteConfigRuleUpdate to not detect non-compliant resources configurations
DeleteConfigurationAggregatorUpdate to not detect non-compliant resources configurations
DeleteConfigurationRecorderResource configuration changes will no longer be recorded
DeleteConformancePackUpdate to not detect non-compliant resources configurations
DeleteDeliveryChannelRemove Config service settings (s3 bucket, sns topic, delivery frequency). Requires StopConfigurationRecorder
DeleteOrganizationConfigRuleUpdate to not detect non-compliant resources configurations
DeleteOrganizationConformancePackUpdate to not detect non-compliant resources configurations
DeleteRemediationConfigurationUpdate to not auto-remediate compromised resource configuration
DeleteRetentionConfigurationUpdate to not retain resource configuration history
PutConfigRuleUpdate to not detect non-compliant resources configurations
PutConfigurationAggregatorUpdate to not detect non-compliant resources configurations
PutConformancePackUpdate to not detect non-compliant resources configurations
PutDeliveryChannelUpdate Config service settings (s3 bucket, sns topic, delivery frequency)
PutOrganizationConfigRuleUpdate to not detect non-compliant resources configurations
PutOrganizationConformancePackUpdate to not detect non-compliant resources configurations
PutRemediationConfigurationsUpdate to not remediate compromised resource configuration, update to execute arbitrary API actions
PutRemediationExceptionsUpdate to not remediate compromised resource configuration
PutRetentionConfigurationUpdate to not retain resource configuration history
StopConfigurationRecorderresource configuration changes will no longer be recorded

To-do:

GuardDuty

GuardDuty is AWS' managed threat detection services. The service evaluates VPC Flow Logs, CloudTrail, and Route53 query logs using signature and machine learning-backed detection methods. There are numerous methods for bypassing detection both through GuardDuty re-configuration and operating within its blind spots.

GuardDuty Disruption

Disable GuardDuty in the Web console translates to the DeleteDetector API call and Pacu's Delete option

Suspend GuardDuty in the Web console translates to the UpdateDetector --no-enable API call and Pacu's Disable option

select *
from cloudtrail_000000000000
where year = '####' and month = '##' and day = '##'
and eventname IN ('CreateFilter','CreateIPSet','CreateSampleFindings','CreateThreatIntelSet',
'DeleteDetector','DeleteMembers','DeletePublishingDestination','DeleteThreatIntelSet',
'DisassociateFromMasterAccount','DisassociateMembers','StopMonitoringMembers',
'UpdateDetector','UpdateFilter','UpdateIPSet','UpdatePublishingDestination','UpdateThreatIntelSet')
order by eventtime desc
ActionImpact
CreateFilterEvate detection. Exempts findings (auto-archive)
CreateIPSetEvate detection. Exempts a potentially malicious IP as trusted
CreateSampleFindingsChaos. Flood GuardDuty with sample findings as a diversion
CreateThreatIntelSetChaos. Flood GuardDuty with false positives as a diversion
DeleteDetectorEvate detection
DeleteMembersEvate detection. Master unaware of member findings. Significant impact if event handling for findings is handled only at the master.
DeletePublishingDestinationDisrupt event flow for threat findings
DeleteThreatIntelSetCustom IP threat list not evaluated
DisassociateFromMasterAccountBypass detection. Master unaware of member findings
DisassociateMembersEvate detection. Master unaware of member findings
StopMonitoringMembersMaster unaware of member findings
UpdateDetectorEvate detection. set -no-enable
UpdateFiltersee CreateFilter
UpdateIPSetsee CreateIPSet
UpdatePublishingDestinationsee DeletePublishingDestination
UpdateThreatIntelSetsee DeleteThreatIntelSet

GuardDuty Recon

select *
from cloudtrail_000000000000
where year = '####' and month = '##' and day = '##'
and eventname IN ('ListMembers','GetMembers',
'ListDetectors','GetDetector',
'ListFilters','GetFilter',
'ListIPSets','GetIPSet',
'ListThreatIntelSets','GetThreatIntelSet')
order by eventtime desc

IAM Access Analyzer

select *
from cloudtrail_000000000000
where year = '####' and month = '##' and day = '##'
and eventname IN ('CreateArchiveRule','DeleteAnalyzer',
'UpdateArchiveRule','UpdateFindings')
order by eventtime desc

Note: organizations.amazonaws.com has an API action for DeregisterDelegatedAdministrator

ActionImpact
CreateArchiveRuleEvade detection. Auto-archive matched findings
DeleteAnalyzerEvade detection. Suppress all findings
UpdateArchiveRuleEvade detection. Auto-archive matched findings
UpdateFindingsEvate detection. Archive sepcific findings

Inspector

https://docs.aws.amazon.com/IAM/latest/UserGuide/list_amazoninspector.html

https://docs.aws.amazon.com/cli/latest/reference/inspector/index.html

select *
from cloudtrail_000000000000
where year = '####' and month = '##' and day = '##'
and eventname IN ('DeleteAssessmentRun','DeleteAssessmentTarget',
'DeleteAssessmentTemplate','UnsubscribeFromEvent','UpdateAssessmentTarget')
order by eventtime desc

Macie(2)

This section applies to the new Macie macie2.amazonaws.com which is not Macie "classic"

select *
from cloudtrail_000000000000
where year = '####' and month = '##' and day = '##'
and eventname IN ('ArchiveFindings','CreateFindingsFilter',
'DeleteMember','DisassociateFromMasterAccount','DisassociateMember',
'DisableMacie',
'UpdateFindingsFilter','UpdateMacieSession','UpdateMemberSession','DisableOrganizationAdminAccount',
'UpdateClassificationJob','UpdateFindingsFilter')
order by eventtime desc

Note that DeregisterDelegatedAdministrator is an eventsource organizations.amazonaws.com

ActionImpact
ArchiveFindingsBypass detection by retroactively removing findings
CreateFindingsFilterBypass detection by setting auto-archive rules (suppress findings)
Organizations:DeregisterDelegatedAdministratorsever delegated admin account from organization master configuration
DeleteMemberBypass detection. After disassociation (still enabled, not reported to master), deleting a member disabled Macie for the Member.
DisassociateFromMasterAccountBypass detection. Macie is still enabled on member, but findings are not reported to master
DisassociateMemberBypass detection. Macie is still enabled on member, but findings are not reported to master
DisableMacieBypass detection. Disables Macie and deletes Macie resources
UpdateFindingsFilterBypass detection by setting auto-archive rules (suppress findings)
UpdateMacieSession"requestParameters": {"status": "PAUSED"}, Bypass decection by suspending Macie or updating Macie configurations
UpdateMemberSession"requestParameters": {"status": "PAUSED"}, Bypass detection by suspending Macie or updating Macie configurations
DisableOrganizationAdminAccountRemove delegated administration account for Macie
UpdateClassificationJobBypass detection by removing scanned resources
UpdateFindingsFilterBypass detection by suppressing findings

EC2

Disrupt VPC Flow Logs

select *
from cloudtrail_000000000000
where year = '####' and month = '##' and day = '##'
and eventsource = 'ec2.amazonaws.com'
and eventname = 'DeleteFlowLogs'

Note: GuardDuty, if enabled, monitors VPC Flow Logs independent of logging configuration. This disruptions primarily impacts custom monitoring solutions and 3rd party software.

ActionImpact
DeleteFlowLogsBypass detection by disabling collection of net flow

S3

Permissions Update

select eventTime, eventSource, eventName, awsRegion, errorCode, errorMessage, userIdentity.arn, sourceIPAddress, requestParameters
from cloudtrail_000000000000
where year = '####' and month = '##' and day = '##'
and eventsource = 's3.amazonaws.com'
and eventname in (
    'PutAccessPointPolicy',
    'PutAccountPublicAccessBlock',
    'PutBucketAcl',
    'PutBucketCORS',
    'PutBucketPolicy',
    'PutBucketPublicAccessBlock',
    'PutObjectAcl'
)
order by eventTime desc

API Actions

ActionTypeImpact
PutAccessPointPolicyaccess permissionsexpand permissions, data exfil
PutAccountPublicAccessBlockaccess permissionsexpand permissions, data exfil
PutBucketAclaccess permissionsexpand permissions, data exfil
PutBucketCORSaccess permissionsexpand permissions, data exfil
PutBucketPolicyaccess permissionsexpand permissions, data exfil
PutBucketPublicAccessBlockaccess permissionsexpand permissions, data exfil
PutObjectAclaccess permissionsexpand permissions, data exfil

Data Management

select eventTime, eventSource, eventName, awsRegion, errorCode, errorMessage, userIdentity.arn, sourceIPAddress, requestParameters
from cloudtrail_000000000000
where year = '####' and month = '##' and day = '##'
and eventsource = 's3.amazonaws.com'
and eventname in (
    'PutBucketLogging',
    'PutBucketWebsite',
    'PutEncryptionConfiguration',
    'PutLifecycleConfiguration',
    'PutReplicationConfiguration',
    'ReplicateObject',
    'RestoreObject'
)
order by eventTime desc

API Actions

ActionTypeImpact
PutBucketLoggingdata managementsuppress logging
PutBucketWebsitedata managementpotential to expose data for exfil
PutEncryptionConfigurationdata managementdisable encryption
PutLifecycleConfigurationdata managementtransfer objects to an accessible target resource, data exfil
PutReplicationConfigurationdata managementtransfer objects to an accessible target resource, data exfil
ReplicateObjectdata managementtransfer objects to an accessible target resource, data exfil
RestoreObjectdata managementaccess potentially sensitve (deleted/archived) data object, secrets access

SecurityHub

References:

Terminology

TermDescription
Insightsaved findings filter
Masterdesignated account for security hub configuration and findings aggregation
Memberaccount associated with a master for findings forwarding and inheritence of configurations
Standardcollection of controls that may be enabled/disabled
Targetactions in Events

API Actions

ActionTypeImpact
BatchDisableStandardsservicesuppression of detection; deletes associated Config Rules
BatchUpdateFindingsfindingsuppression of findings
DeleteActionTargetservicesuppression of alerting
DeleteInsightfindingsuppression of existing findings filter
DeleteMembersservicesever master-member reporting to suppress findings alerting
DisableImportFindingsForProductservicesuppress findings from source detection product
DisableSecurityHubservicesuppression of detection and alerting
DisassociateFromMasterAccountservicesever master-member reporting to suppress findings alerting
DisassociateMembersservicesever master-member reporting to suppress findings alerting
UpdateActionTargetservicesuppression of alerting
UpdateFindingsfindingsuppression of existing findings
UpdateInsightfindingsee DeleteInsight
UpdateStandardsControlservicesuppression of findings detection

SecurityHub Service Disruption

select *
from cloudtrail_000000000000
where year = '####' and month = '##' and day = '##'
and eventsource = 'securityhub.amazonaws.com'
and eventname in ('BatchDisableStandards',
'DeleteActionTarget','DeleteMembers',
'DisableImportFindingsForProduct','DisableSecurityHub',
'DisassociateFromMasterAccount','DisassociateMembers',
'UpdateActionTarget','UpdateStandardsControl')

SecurityHub Findings Disruption

select *
from cloudtrail_000000000000
where year = '####' and month = '##' and day = '##'
and eventsource = 'securityhub.amazonaws.com'
and eventname in ('BatchUpdateFindings',
'DeleteInsight',
'UpdateFindings','UpdateInsight')

Web Application Firewall (WAF)

select *
from cloudtrail_000000000000
where (year = '####' and month = '##' and day = '##')
and eventname in ('DeleteFirewallManagerRuleGroups','DeleteIPSet',
'DeleteLoggingConfiguration','DeletePermissionPolicy','DeleteRegexPatternSet',
'DeleteRuleGroup','DeleteWebACL','DisassociateWebACL',
'PutLoggingConfiguration','PutPermissionPolicy',
'UpdateIPSet','UpdateRegexPatternSet','UpdateRuleGroup','UpdateWebACL')
ActionImpact
DeleteFirewallManagerRuleGroupsrexpand network access; if not managed by Firewall manager
DeleteIPSetexpand network access
DeleteLoggingConfigurationdisrupt logging
DeletePermissionPolicyexpand network access
DeleteRegexPatternSetexpand network access
DeleteRuleGroupexpand network access
DeleteWebACLexpand network access
DisassociateWebACLexpand network access
PutLoggingConfigurationsee DeleteLoggingConfiguration
PutPermissionPolicyexpand network access
UpdateIPSetexpand network access - allow attack network
UpdateRegexPatternSetexpand network access
UpdateRuleGroupexpand network access
UpdateWebACLexpand network access

Network Flow Analysis

This is a proof-of-concept description. Expected changes for an enterprise implementation include:

Pre-requisites include:

Additional query capabilities are introduced by enriching Flow Logs with additional metadata.

Reference: https://aws.amazon.com/blogs/aws/learn-from-your-vpc-flow-logs-with-additional-meta-data/

Manual Setup

Reference: https://docs.aws.amazon.com/athena/latest/ug/vpc-flow-logs.html

Add table to Athena

CREATE EXTERNAL TABLE IF NOT EXISTS vpc_flow_logs_000000000000 (
  version int,
  account string,
  interfaceid string,
  sourceaddress string,
  destinationaddress string,
  sourceport int,
  destinationport int,
  protocol int,
  numpackets int,
  numbytes bigint,
  starttime int,
  endtime int,
  action string,
  logstatus string
)
PARTITIONED BY (`date` date)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ' '
LOCATION 's3://{bucket_name}/AWSLogs/000000000000/vpcflowlogs/{region}/'
TBLPROPERTIES ("skip.header.line.count"="1");

Add date partition

Partitions limits the query return set for speed and cost savings.

ALTER TABLE vpc_flow_logs_000000000000
ADD PARTITION (`date`='yyyy-mm-dd')
location 's3://bucket_name}/AWSLogs/000000000000/vpcflowlogs/{region}/yyyy/mm/dd';

Useful Queries

General concepts

Packet count proves useful for surfacing a high volume of traffic on protocols with smaller payload.

measurementquery
packet countsum(numpackets)
traffic volumesum(numbytes)

Inbound Traffic

Accepted packets by port

This is the traffic (packet count) allowed to reach your resource destinationaddress is the private IP of your resource

select destinationport, sum(numpackets) as packets
from vpc_flow_logs_000000000000
WHERE date = DATE('yyyy-mm-dd')
and destinationaddress = 'x.x.x.x'
and action = 'ACCEPT'
group by destinationport
order by packets desc
limit 10

Rejected packets by port

This is the traffic (packet count) blocked from reaching your resource. REJECTED traffic may be due to a NACL Deny rule or lack of a security group rule. destinationaddress is the private IP of your resource

select destinationport, sum(numpackets) as packets
from vpc_flow_logs_000000000000
WHERE date = DATE('yyyy-mm-dd')
and destinationaddress = 'x.x.x.x'
and action = 'REJECT'
group by destinationport
order by packets desc
limit 10

Volume by sourceaddress

destinationaddress is the private IP of your resource

select sourceaddress, sum(numbytes) as volume
from vpc_flow_logs_000000000000
WHERE date = DATE('yyyy-mm-dd')
and destinationaddress = 'x.x.x.x'
group by sourceaddress
order by volume desc
limit 10

Volume by destinationport

destinationaddress is the private IP of your resource

select destinationport, sum(numbytes) as volume
from vpc_flow_logs_000000000000
WHERE date = DATE('yyyy-mm-dd')
and destinationaddress = 'x.x.x.x'
group by destinationport
order by volume desc
limit 10

Assess specific time range

destinationaddress is the private IP of your resource starttime and endtime uses epoch

Reference: https://www.epochconverter.com/

select *
from vpc_flow_logs_000000000000
WHERE date = DATE('yyyy-mm-dd') 
and destinationaddress = 'x.x.x.x'
and starttime > 1596225600
and endtime < 1596226200

Outbound Traffic

Volume by desination

sourceaddress is the private IP of your resource

select destinationaddress, sum(numbytes) as volume
from vpc_flow_logs_000000000000
WHERE date = DATE('yyyy-mm-dd')
and sourceaddress = 'x.x.x.x'
group by destinationaddress
order by volume desc
limit 10

Volume by port

destinationaddress is the private IP of your resource

select destinationport, sum(numbytes) as volume
from vpc_flow_logs_000000000000
WHERE date = DATE('yyyy-mm-dd')
and sourceaddress = 'x.x.x.x'
group by destinationport
order by volume desc
limit 10

Assess specific time range

source is the private IP of your resource starttime and endtime uses epoch

Reference: https://www.epochconverter.com/

select *
from vpc_flow_logs_000000000000
WHERE date = DATE('yyyy-mm-dd') 
and sourceaddress = 'x.x.x.x'
and starttime > 1596225600
and endtime < 1596226200

Rejected by port

This is the traffic (packet count) allowed to reach your resource destinationaddress is the private IP of your resource

select destinationport, sum(numpackets) as packets
from vpc_flow_logs_000000000000
WHERE date = DATE('yyyy-mm-dd')
and destinationaddress = 'x.x.x.x'
and action = 'ACCEPT'
group by destinationport
order by packets desc
limit 10

Rejected by destination

select destinationaddress, sum(numpackets) as packets
from vpc_flow_logs_000000000000
WHERE date = DATE('yyyy-mm-dd')
and sourceaddress = 'x.x.x.x'
and action = 'REJECT'
group by destinationaddress
order by packets desc
limit 10

Example: Connectionless LDAP Reflection Attack

destinationaddress is your resource being used in the attack protocol is 17/UDP destinationport is 389/LDAP

select sourceaddress, sum(numpackets) as packets
from vpc_flow_logs_000000000000
WHERE date = DATE('yyyy-mm-dd')
and destinationaddress = 'x.x.x.x'
and protocol = 17
and destinationport = 389
group by sourceaddress
order by packets desc

Useful CloudTrail fields

References

KeyValues / Notes
useridentity.arnarn:aws:sts::000000000000:assumed-role/AssumedRoleName/AssumedRoleSessionName
useridentity.accesskeyidAKIA*****************
useridentity.sessioncontext.attributes.mfaauthenticatedtrue, false, null
useridentity.sessioncontext.sessionissuer.typeRole
useridentity.sessioncontext.sessionissuer.arnarn:aws:iam::123456789012:role/RoleToBeAssumed
useridentity.sessioncontext.sessionissuer.usernameRoleToBeAssumed
useridentity.principalidAROAxxxxxxxxxxxxxxxxx:role-session-name (AIDAxxxxxxxxxxxxxxxxx, AROAxxxxxxxxxxxxxxxxx, saml:namequalifier and saml:sub keys)
useridentity.accountididentifies access from external accounts
useridentity.typeAssumedRole, AWSService, Unknown, IAMUser, AWSAccount, SAMLUser, Root
eventtimedate and time of request in UTC
eventsourceAWS service name xxx.amazonaws.com
eventnamethe API action
awsregione.g. us-east-1
sourceipaddressx.x.x.x or xxx.amazonaws.com
useragente.g. Botocore/1.13.43 Python/3.7.5 Linux/3.10.0-1127.13.1.el7.x86_64
errorcodemay alternatively be in responseElements
errormessagemay alternatively be in responseElements
requestparametersdetailed parameters for the API action
responseElementsfor create,update,delete actions
eventtypeAwsApiCall, AwsServiceEvent, AwsConsoleSignin
eventidglobally unique CloudTrail event ID, worth remembering for easier retrieval of valuable events

Credit and References