Home

Awesome

Sloop - Kubernetes History Visualization

Publish Status Build Status Go Report Card

<img src="https://github.com/salesforce/sloop/raw/master/other/sloop_logo_color_small_notext.png">

Sloop monitors Kubernetes, recording histories of events and resource state changes and providing visualizations to aid in debugging past events.

Key features:

  1. Allows you to find and inspect resources that no longer exist (example: discover what host the pod from the previous deployment was using).
  2. Provides timeline displays that show rollouts of related resources in updates to Deployments, ReplicaSets, and StatefulSets.
  3. Helps debug transient and intermittent errors.
  4. Allows you to see changes over time in a Kubernetes application.
  5. Is a self-contained service with no dependencies on distributed storage.

Screenshots

Screenshot1

Architecture Overview

Architecture

Install

Sloop can be installed using any of these options:

Helm Chart

Users can install sloop by using helm chart now, for instructions refer helm readme

Precompiled Binaries

TODO: See the Releases.

Build from Source

Building Sloop from source needs a working Go environment with the version defined in the go.mod file or greater.

See: https://golang.org/doc/install

Clone the sloop repository and build using make:

mkdir -p $GOPATH/src/github.com/salesforce
cd $GOPATH/src/github.com/salesforce
git clone https://github.com/salesforce/sloop.git
cd sloop
make
$GOPATH/bin/sloop

When complete, you should have a running Sloop version accessing the current context from your kubeConfig. Just point your browser at http://localhost:8080/

Other makefile targets:

Local Docker Run

To run from Docker you need to host mount your kubeconfig:

make docker-snapshot
docker run --rm -it -p 8080:8080 -v ~/.kube/:/kube/ -e KUBECONFIG=/kube/config sloop

In this mode, data is written to a memory-backed volume and is discarded after each run. To preserve the data, you can host-mount /data with something like -v /data/:/some_path_on_host/

Updating webfiles folder

To reflect any changes to webserver/webfiles, run the following command on terminal while within the webserver directory before submitting a pr:

go-bindata -pkg webserver -o bindata.go webfiles/

This will update the bindata folder with your changes to any html, css or javascript files within the directory.

Local Docker Run and connecting to EKS

This is very similar to above but abstracts running docker with AWS credentials for connecting to EKS

make docker
export AWS_ACCESS_KEY_ID=<access_key_id> AWS_SECRET_ACCESS_KEY=<secret_access_key> AWS_SESSION_TOKEN=<session_token>
./providers/aws/sloop_to_eks.sh <cluster name>

Data retention policy stated above still applies in this case.

Backup & Restore

This is an advanced feature. Use with caution.

To download a backup of the database, navigate to http://localhost:8080/data/backup

To restore from a backup, start sloop with the -restore-database-file flag set to the backup file downloaded in the previous step. When restoring, you may also wish to set the -disable-kube-watch=true flag to stop new writes from occurring and/or the -context flag to restore the database into a different context.

Memory Consumption

Sloop's memory usage can be managed by tweaking several options:

Apart from these flags some other values can be tweaked to fit in the memory constraints. Following are some examples of setups.

               // 0.5<<20 (524288 bytes = 0.5 Mb)
               "badger-max-table-size=524288",
               "badger-number-of-compactors=1",
               "badger-number-of-level-zero-tables=1",
               "badger-number-of-zero-tables-stall=2",
               // 16<<20 (16777216 bytes = 16 Mb)
               "badger-max-table-size=16777216",
               "badger-number-of-compactors=1",
               "badger-number-of-level-zero-tables=1",
               "badger-number-of-zero-tables-stall=2",
               // 32<<20 (33554432 bytes = 32 Mb)
               "badger-max-table-size=33554432",
               "badger-number-of-compactors=1",
               "badger-number-of-level-zero-tables=2",
               "badger-number-of-zero-tables-stall=3",

Apart from the above settings, max-disk-mb and max-look-back can be tweaked according to input data and memory constraints.

Prometheus

Sloop uses the Prometheus library to emit metrics, which is very helpful for performance debugging.

In the root of the repo is a Prometheus config file prometheus.yml.

On OSX you can install Prometheus with brew install prometheus. Then start it from the sloop directory by running prometheus

Open your browser to http://localhost:9090.

An example of a useful query is rate(kubewatch_event_count[5m])

Event filtering

Events can be excluded from Sloop by adding exclusionRules to the config file:

{
  "defaultNamespace": "default",
  "defaultKind": "Pod",
  "defaultLookback": "1h",
  [...]
  "exclusionRules": {
    "_all": [
      {"==": [ { "var": "metadata.namespace" }, "kube-system" ]}
    ],
    "Pod": [
      {"==": [ { "var": "metadata.name" }, "sloop-0" ]}
    ],
    "Job": [
      {"in": [ { "var": "metadata.name" }, [ "cron1", "cron3" ] ]}
    ]
  }
}`

Adding rules can help to reduce resources consumed by Sloop and remove unwanted noise from the UI for events that are of no interest.

Limiting rules to specific kinds

Rule format and supported operations

Rules should follow the JsonLogic format and are evaluated against the json representation of the Kubernetes API object related to the event (see below).

Available operators, such as == and in shown above, are documented here.

Data available to rule logic

Kubernetes API conventions for objects require the following keys to exist in the json data for all resources, all of which can be referenced in rules:

Some commonly useful fields under the metadata object are:

Type specific data

Some resources contain additional type-specific fields, for example PersistentVolumeClaimSpec objects have fields named selector and storageClassName.

Type specific fields for each object and their corresponding keys in the object json representation are documented in the core API, e.g. for PersistentVolumeClaimSpec objects the documentation is here.

Contributing

Refer to CONTRIBUTING.md<br>

License

BSD 3-Clause