Home

Awesome

Build Status Maven Central Javadoc

AWSSSMChaosRunner

AWSSSMChaosRunner is a library which simplifies failure injection testing and chaos engineering for EC2 and ECS (with EC2 launch type). It offers the following options for failure injection -

An in-depth introduction to this library and how Prime Video uses it can be found here - https://aws.amazon.com/blogs/opensource/building-resilient-services-at-prime-video-with-chaos-engineering/

Usage with AWS Systems Manager SendCommand

  1. Setup permissions for calling SSM from tests package

    This can be done in many different ways. The approach described here generates temporary credentials for AWS SSM on each run of the tests. To enable this the following are needed

    • An IAM role with the following permissions. (JSON snippet) 
      {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "sts:AssumeRole",
                "ssm:CancelCommand",
                "ssm:CreateDocument",
                "ssm:DeleteDocument",
                "ssm:DescribeDocument",
                "ssm:DescribeInstanceInformation",
                "ssm:DescribeDocumentParameters",
                "ssm:DescribeInstanceProperties",
                "ssm:GetDocument",
                "ssm:ListTagsForResource",
                "ssm:ListDocuments",
                "ssm:ListDocumentVersions",
                "ssm:SendCommand"
            ],
            "Resource": [
                "*"
            ],
            "Effect": "Allow"
        },
        {
            "Action": [
                "ec2:DescribeInstances",
                "iam:PassRole",
                "iam:ListRoles"
            ],
            "Resource": [
                "*"
            ],
            "Effect": "Allow"
        },
        {
            "Action": [
                "ssm:StopAutomationExecution",
                "ssm:StartAutomationExecution",
                "ssm:DescribeAutomationExecutions",
                "ssm:GetAutomationExecution"
            ],
            "Resource": [
                "*"
            ],
            "Effect": "Allow"
        }
    ]
}
    • An IAM user which can assume the above role.

  2. Add AWSSSMChaosRunner maven dependency to your tests package

    <dependency>
  <groupId>software.amazon.awsssmchaosrunner</groupId>
  <artifactId>awsssmchaosrunner</artifactId>
  <version>1.3.0</version>
</dependency>

  3. Initialise the SSM Client (Kotlin snippet)

    @Bean
open fun awsSecurityTokenService(
   credentialsProvider: AWSCredentialsProvider, 
   awsRegion: String
   ): AWSSecurityTokenService {
    return AWSSecurityTokenServiceClientBuilder.standard()
        .withCredentials(credentialsProvider)
        .withRegion(awsRegion)
        .build()
}

@Bean
open fun awsSimpleSystemsManagement(
   securityTokenService: AWSSecurityTokenService,
   awsAccountId: String,
   chaosRunnerRoleName: String
   ): AWSSimpleSystemsManagement {
    val chaosRunnerRoleArn = "arn:aws:iam::$awsAccountId:role/$chaosRunnerRoleName"
    val credentialsProvider = STSAssumeRoleSessionCredentialsProvider
        .Builder(chaosRunnerRoleArn, "ChaosRunnerSession")
        .withStsClient(securityTokenService).build()

    return AWSSimpleSystemsManagementClientBuilder.standard()
        .withCredentials(credentialsProvider)
        .build()
}

  4. Start the fault injection attack before starting the test and stop it after the test (Kotlin snippet)

    import software.amazon.awsssmchaosrunner.attacks.SSMAttack
import software.amazon.awsssmchaosrunner.attacks.SSMAttack.Companion.getAttack
...

@Before
override fun initialise(args: Array<String>) {
    if (shouldExecuteChaosRunner()) {
        ssm = applicationContext.getBean(AWSSimpleSystemsManagement::class.java)
        ssmAttack = getAttack(ssm, attackConfiguration)
        command = ssmAttack.start()
    }
}

@After
override fun destroy() {
    ssmAttack.stop(command)
}

  5. Run the test

Usage with AWS FIS

  1. Setup permissions for calling SSM from tests package

    This can be done in many different ways. The approach described here generates temporary credentials for AWS SSM on each run of the tests. To enable this the following are needed

    • An IAM role with the following permissions. (JSON snippet) 
      {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "sts:AssumeRole",
                "ec2:DescribeInstances",
                "iam:ListRoles",
                "ssm:ListCommands",
                "ssm:SendCommand",
                "ssm:CancelCommand",
                "iam:PassRole",
                "ec2:RebootInstances",
                "ec2:StopInstances",
                "ec2:StartInstances",
                "ec2:TerminateInstances",
                "fis:InjectApiInternalError",
                "fis:InjectApiThrottleError",
                "fis:InjectApiUnavailableError",
                "fis:ListExperimentTemplates",
                "fis:ListActions",
                "fis:ListTargetResourceTypes",
                "fis:ListExperiments",
                "fis:GetTargetResourceType",
                "fis:CreateExperimentTemplate",
                "fis:DeleteExperimentTemplate",
                "fis:StopExperiment",
                "fis:StartExperiment"                    
            ],
            "Resource": [
                "*"
            ],
            "Effect": "Allow"
        },
        {
            "Action": [
                "iam:CreateServiceLinkedRole"
            ],
            "Resource": [
                "*"
            ],
            "Effect": "Allow",
            "Conditions": {
                  "StringEquals": {
                       "iam:AWSServiceName": "fis.amazonaws.com"
                  }
            }
        }
    ]
}
    • An IAM user which can assume the above role.

  2. Add AWSSSMChaosRunner maven dependency to your tests package

    <dependency>
  <groupId>software.amazon.awsssmchaosrunner</groupId>
  <artifactId>awsssmchaosrunner</artifactId>
  <version>1.3.0</version>
</dependency>

  3. Initialise the FIS Client (Kotlin snippet)

    //Java code snippet
String executionRoleArn = getenv("EXECUTION_ROLE_ARN");
Region awsRegion = Region.of(getenv("AWS_REGION"));

StsClient stsClient = StsClient.builder().build();
StsAssumeRoleCredentialsProvider assumeRoleCredentialsProvider = StsAssumeRoleCredentialsProvider.builder()
       .refreshRequest(AssumeRoleRequest.builder()
                .roleArn(executionRoleArn)
                .roleSessionName("ChaosRunnerSession")
                .build())
                .stsClient(stsClient)
                .build();
FisClient fisClient = FisClient.builder().credentialsProvider(assumeRoleCredentialsProvider).region(awsRegion).build();

  4. Configure and execute the FIS failure injection

    String targetsSelectionMode = "ALL";
String cloudWatchLogGroupArn = "";
String stopConditionCloudWatchAlarmArn = "";
String name = "IOStress"; // This failure injection consumes disk space
String duration = "PT2M";
Map<String, String> otherFailureInjectionParameters = Collections.emptyMap();

FISAttack.Companion.AttackConfiguration attackConfiguration = new FISAttack.Companion.AttackConfiguration(targets,
        targetsSelectionMode,
        cloudWatchLogGroupArn,
        stopConditionCloudWatchAlarmArn,
        executionRoleArn);
FISSendCommandAttack.Companion.ActionConfiguration actionConfiguration = new FISSendCommandAttack.Companion.ActionConfiguration(
        name,
        duration,
        awsRegion.toString(),
        otherFailureInjectionParameters
);
FISAttack fisAttack = FISSendCommandAttack.Companion.getAttack(fisClient, attackConfiguration, actionConfiguration);
StartExperimentResponse experiment = fisAttack.start();
...
...
boolean deleteExperimentTemplate = false;
fisAttack.stop(experiment, deleteExperimentTemplate);

FAQs