Home

Awesome

CF Acceptance Tests (CATs)

This suite exercises a Cloud Foundry deployment using the cf CLI and curl. It is scoped to testing user-facing, end-to-end features.

For example, one test pushes an app with cf push, hits an endpoint on the app with curl that causes it to crash, and asserts that we see a crash event in cf events.

Tests that won't be introduced here include things like basic CRUD of an object in the Cloud Controller. Such tests belong with the component they relate to.

These tests are not intended for use against production systems. They're meant for acceptance environments used by people developing Cloud Foundry's releases. While these tests attempt to clean up after themselves, there's no guarantee that they won't change the state of your system in an undesirable way. For lightweight system tests that are safe to run against a production environment, please use the CF Smoke Tests.

NOTE: Because we want to parallelize execution, tests should be written to be executable independently. Tests should not depend on state in other tests, and should not modify the CF state in such a way as to impact other tests.

  1. Test Setup
    1. Install Required Dependencies
    2. Test Configuration
  2. Test Execution
  3. Explanation of Test Groups
  4. Contributing

Test Setup

Prerequisites for running CATS

Updating go dependencies

All go dependencies required by CATs are vendored in the vendor directory.

Make sure to have Golang 1.22+

In order to update a current dependency to a specific version, do the following:

cd cf-acceptance-tests
source .envrc
go get <import_path>@<revision_number>
go mod vendor

If you'd like to add a new dependency just run:

go mod tidy
go mod vendor

Test Configuration

You must set the environment variable $CONFIG which points to a JSON file that contains several pieces of data that will be used to configure the acceptance tests, e.g. telling the tests how to target your running Cloud Foundry deployment and what tests to run.

The following can be pasted into a terminal and will set up a sufficient $CONFIG to run the core test suites against a BOSH-Lite deployment of CF.

See example-cats-config.json for reference.

Only the following test groups are run by default:

include_apps
include_detect
include_routing
include_v3
include_app_syslog_tcp

The full set of config parameters is explained below:

Required parameters:
Optional parameters:

include_* parameters are used to specify whether to skip tests based on how a deployment is configured.

Buildpack Names

Many tests specify a buildpack when pushing an app, so that on diego the app staging process completes in less time. The default names for the buildpacks are as follows; if you have buildpacks with different names, you can override them by setting different names:

For the Cloud Native Buildpacks lifecycle, you can override them by setting different names:

Route Services Test Group Setup

The route_services test group pushes applications which must be able to reach the load balancer of your Cloud Foundry deployment. This requires configuring application security groups to support this. Your deployment manifest should include the following data if you are running the route_services group:

...
properties:
  ...
  cc:
    ...
    security_group_definitions:
      - name: load_balancer
        rules:
        - protocol: all
          destination: IP_OF_YOUR_LOAD_BALANCER # (e.g. 10.244.0.34 for a standard deployment of Cloud Foundry on BOSH-Lite)
    default_running_security_groups: ["load_balancer"]

Container Networking and Application Security Groups

To run tests that exercise container networking and running application security groups, the include_security_groups flags must be true.

The Windows ASG tests require an unallocated IP on the private network used by the CF deployment, set with the unallocated_ip_for_security_group config value. Environments created by bbl on public clouds can use the default value of 10.0.244.255. vSphere and Openstack environments may require a custom IP.

Private Docker

To run tests that exercise the use of credentials to access a private docker registry, the include_private_docker_registry flag must be true, and the following config values must be provided:

These tests assume that the specified private docker image is a private version of the cloudfoundry/diego-docker-app:latest. To upload a private version to your DockerHub account, first create a private repository on DockerHub and log in to docker on the command line. Then run the following commands:

docker pull cloudfoundry/diego-docker-app:latest
docker tag cloudfoundry/diego-docker-app:latest <your-private-repo>:<some-tag>
docker push <your-private-repo>:<some-tag>

The value for the private_docker_registry_image config value in this case would be "<your-private-repo>:<some-tag>".

Routing Isolation Segments

To run tests that involve routing isolation segments, the following config values must be provided:

Read the documentation here for further setup details.

Credhub Modes

Capturing Test Output

When a test fails, look for the test group name ([services] in the example below) in the test output:

• Failure in Spec Setup (BeforeEach) [34.662 seconds]
[services] Service Instance Lifecycle

If you set a value for artifacts_directory in your $CONFIG file, then you will be able to capture cf trace output from failed test runs, this output may be useful in cases where the normal test output is not enough to debug an issue. The cf trace output for the tests in these specs will be found in CF-TRACE-Applications-*.txt in the artifacts_directory.

Test Execution

To execute tests according to your configuration, run the bin/test script with $CONFIG set to your integration_config.json file path.

Parallel execution

It's possible to execute all test groups, and have tests run in parallel across multiple processes. This parallelization can significantly decrease CATs runtime.

However, be careful with this number, as it's effectively "how many apps to push at once", as nearly every example pushes an app.

Use the --procs flag to set the number of parallel processes, e.g.:

./bin/test --procs=12

For reference here's how many parallel processes the Release Integration team runs with:

Foundation Type# of Parallel Processes
Vanilla CF12
BOSH Lite4
Focusing Test Groups

If you are already familiar with CATs you probably know that there are many test groups. You may not wish to run all the tests in all contexts, and sometimes you may want to focus individual test groups to pinpoint a failure. To execute a specific group of acceptance tests, e.g. routing/, edit your integration_config.json file and set all include_* values to false except for include_routing then run the following:

./bin/test

To execute tests in a single file use an FDescribe block around the tests in that file:

var _ = AppsDescribe("Apps", func() {
  FDescribe("Focused tests", func() { // Add this line here
  // ... rest of file
  }) // Close here
})

The test group names correspond to directory names.

Verbose Output

To see verbose output from ginkgo, use the -v flag.

./bin/test -v

You can of course combine the -v flag with the --procs=N flag.

Overall Test Timeout Setting

From Ginkgo 2.0, the default timeout for the entire test has been changed to 1 hour (refer to: Ginkgo 2.0 Migration Guide - Timeout Behavior).

Tests may run longer than an hour depending on their environment and parallelism, with the possibility of failure.

Adjust the test timeout as needed with the --timeout flag:

./bin/test --timeout=24h

For individual timeouts, such as the timeout for cf push, refer to Test Configuration.

Explanation of Test Groups

Test Group NameDescription
app_syslog_tcpTests the ability to configure an app syslog drain listener.
appsTests the core functionalities of Cloud Foundry: staging, running, logging, routing, buildpacks, etc. This test group should always pass against a sound Cloud Foundry deployment.
credhubTests CredHub-delivered Secure Service credentials in the service binding. CredHub configuration is required to run these tests. In addition to selecting a credhub_mode, credhub_client and credhub_secret values are required for these tests.
cnbTests our ability to use cloud native buildpacks.
detectTests the ability of the platform to detect the correct buildpack for compiling an application if no buildpack is explicitly specified.
dockerTests our ability to run docker containers on Diego and that we handle docker metadata correctly.
internet_dependentTests the feature of being able to specify a buildpack via a Github URL. As such, this depends on your Cloud Foundry application containers having access to the Internet. You should take into account the configuration of the network into which you've deployed your Cloud Foundry, as well as any security group settings applied to application containers.
isolation_segmentsThis test group requires that Diego be deployed with a minimum of 2 cells. One of those cells must have been deployed with a placement_tag. If the deployment has been deployed with a routing isolation segment, isolation_segment_domain must also be set. For more information, please refer to the Isolation Segments documentation.
route_servicesTests the Route Services feature of Cloud Foundry.
routingThis package contains routing specific acceptance tests (context paths, wildcards, SSL termination, sticky sessions, and zipkin tracing).
routing_isolation_segmentsTests that requests to isolated apps are only routed through isolated routers, and vice versa. It requires all of the setup for the isolation segments test suite. Additionally, a minimum of two Gorouter instances must be deployed. One instance must be configured with the property routing_table_sharding_mode: shared-and-segments. The other instance must have the properties routing_table_sharding_mode: segments and isolation_segments: [YOUR_PLACEMENT_TAG_HERE]. The isolation_segment_name in the CATs properties must match the placement_tag and isolation_segment.isolation_segment_domain must be set and traffic to that domain should go to the isolated router.
security_groupsTests the Security Groups feature of Cloud Foundry.
service_discoveryTests the Service Discovery feature for applications running on Cloud Foundry.
servicesTests various features related to services, e.g. registering a service broker via the service broker API. Some of these tests exercise special integrations, such as Single Sign-On authentication; you may wish to run some tests in this package but selectively skip others if you haven't configured the required integrations (by setting include_sso parameter to falsein your configuration).
sshTests communication with Diego apps via ssh, scp, and sftp.
tasksTests Cloud Foundry's Tasks feature.
tcp_routingTests TCP Routing Feature of Cloud Foundry. You need to make sure you've set up a TCP domain tcp.<SYSTEM_DOMAIN> as described here. If you are using bbl (BOSH Bootloader), TCP domain is set up for you automatically.
user_provided_servicesTests features related to creating and binding user-provided services for holding app credentials securely.
v3This test group contains tests for the next-generation v3 Cloud Controller API.
volume_servicesTests the Volume Services feature of Cloud Foundry.

Contributing

This repository uses go mod to manage go dependencies.

All go dependencies required by CATs are vendored in the vendor directory.

When making changes to the test suite that bring in additional go packages, you should use the following workflow:

If you can use the most recent version of a dependency use go mod tidy, otherwise use go get <dependency>@<version>. Both of these require go modules to be enabled via the envrc. Finally use go mod vendor to add the dependencies to the vendor directory.

For tools and assets, please use the helpers/assets/tools.go via the go mod tool workflow

For additional information refer to the official wiki and the official examples repo

Although the default branch for this repository is main, we ask that all pull requests be made against the develop branch. Please run the unit tests and make sure they are passing before submitting. Use ./bin/run_units to run these unit tests.

Note: it is necessary to run the tests from the root of the repo.

Code Conventions

There are a number of conventions we recommend developers of CF acceptance tests adopt:

  1. When pushing an app:

For example:

Expect(cf.Cf("push", appName,
    "-b", buildpackName,                  // specify buildpack
    "-m", DEFAULT_MEMORY_LIMIT,           // specify memory limit
    "-d", Config.AppsDomain,              // specify app domain
).Wait(Config.CfPushTimeoutDuration())).To(Exit(0))

  1. Delete all resources that are created, e.g. apps, routes, quotas, etc. This is in order to leave the system in the same state it was found in. For example, to delete apps and their associated routes:

    	Expect(cf.Cf("delete", myAppName, "-f", "-r").Wait()).To(Exit(0))

  2. Specifically for apps, before tearing them down, print the app guid and recent application logs. There is a helper method AppReport provided in the app_helpers package for this purpose.

    AfterEach(func() {
  app_helpers.AppReport(appName)
})

  3. Document the purpose of your test groups in this repo's README.md. This is especially important when changing the explicit behavior of existing test groups or adding new test groups.

  4. Document all changes to the config object in this repo's README.md.

  5. If you add a test that requires a new minimum cf CLI version, update the minCliVersion in cats_suite_test.go .