Awesome

Kubespot (AWS)

AWS EKS Setup for PCI-DSS, SOC2, HIPAA

Kubespot is AWS EKS customized to add security postures around SOC2, HIPAA, and PCI compliance. It is distributed as an open source terraform module allowing you to run it within your own AWS account without lock-in. Kubespot has been developed over a half a decade evolving with the AWS EKS distribution and before that kops. It is in use within multiple startups that have scaled from a couple founders in an apartment to billion dollar unicorns. By using Kubespot they were able to achieve the technical requirements for compliance while being able to deploy software fast.

Kubespot is a light wrapper around AWS EKS. The primary changes included in Kubespot are:

• Locked down with security groups, private subnets and other compliance related requirements.
• Locked down RDS and Elasticache if needed.
• Users have a single Load Balancer through which all requests go through to reduce costs.
• KEDA is used for scaling on event metrics such as queue sizes, user requests, CPU, memory or anything else Keda supports.
• Karpenter is used for autoscaling.
• Instance are lockdown with encryption, and a regular node cycle rate is set.

Tools & Setup

brew install kubectl kubernetes-helm awscli terraform

Cluster Usage

If the infrastructure is using the opsZero infrastructure as code template then you access the resources like the following:

Add your IAM credentials in ~/.aws/credentials.

[profile_name]
aws_access_key_id=<>key>
aws_secret_access_key=<secret_key>
region=us-west-2

cd environments/<nameofenv>
make kubeconfig
export KUBECONFIG=./kubeconfig # add to a .zshrc
kubectl get pods

Autoscaler

Kubespot uses Karpenter as the default autoscaler. To configure the autoscaler we need to create a file like the one below and run:

kubectl apply -f karpenter.yml

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      requirements:
        - key: "karpenter.k8s.aws/instance-category"
          operator: In
          values: ["t", "c", "m"]
        - key: "kubernetes.io/arch"
          operator: In
          values: ["amd64"]
        - key: "karpenter.k8s.aws/instance-cpu"
          operator: In
          values: ["1", "2", "4", "8", "16"]
        - key: "karpenter.k8s.aws/instance-hypervisor"
          operator: In
          values: ["nitro"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
      nodeClassRef:
        name: default
  disruption:
    consolidationPolicy: WhenUnderutilized
    expireAfter: 2h # 30 * 24h = 720h
---
apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
  name: default
spec:
  amiFamily: Bottlerocket # Amazon Linux 2
  role: "Karpenter-opszero" # Set the name of the cluster
  subnetSelectorTerms:
    - tags:
        Name: opszero-public
  securityGroupSelectorTerms:
    - tags:
        Name: eks-cluster-sg-opszero-1249901478

Knative

brew install knative/client/kn
brew tap knative-extensions/kn-plugins

kubectl apply -f https://github.com/knative/serving/releases/download/knative-v1.13.1/serving-crds.yaml
kubectl apply -f https://github.com/knative/serving/releases/download/knative-v1.13.1/serving-core.yaml
kubectl apply -f https://github.com/knative/net-kourier/releases/download/knative-v1.13.0/kourier.yaml

kubectl patch configmap/config-network --namespace knative-serving --type merge --patch '{"data":{"ingress-class":"kourier.ingress.networking.knative.dev"}}'
kubectl patch configmap/config-domain --namespace knative-serving --type merge --patch '{"data":{"fn.opszero.com":""}}'

kubectl apply -f https://github.com/knative/serving/releases/download/knative-v1.13.1/serving-hpa.yaml
kubectl apply -f https://github.com/knative/net-certmanager/releases/download/knative-v1.13.0/release.yaml

kubectl edit configmap config-network -n knative-serving
# Turn the tls
#data:
#  external-domain-tls: Enabled
  http-protocol: Redirected

kubectl edit --namespace knative-serving configmap config-network

namespace-wildcard-cert-selector:
  matchExpressions:
    - key: "kubernetes.io/metadata.name"
      operator: "In"
      values: ["my-namespace", "my-other-namespace"]


kubectl edit configmap config-certmanager -n knative-serving

# apiVersion: v1
# kind: ConfigMap
# metadata:
#   name: config-certmanager
#   namespace: knative-serving
#   labels:
#     networking.knative.dev/certificate-provider: cert-manager
# data:
#   issuerRef: |
#     kind: ClusterIssuer
#     name: letsencrypt-http01-issuer

Apply the following:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-http01-issuer
spec:
  acme:
    privateKeySecretRef:
      name: letsencrypt
    server: https://acme-v02.api.letsencrypt.org/directory
    solvers:
    - http01:
       ingress:
         class: kourier.ingress.networking.knative.dev

Cluster Setup

aws iam create-service-linked-role --aws-service-name spot.amazonaws.com

CIS Kubernetes Benchmark

Note: PodSecurityPolicy (PSP) is deprecated and PodSecurity admission controller is the new standard. The CIS Benchmark is still using PSP. We have converted the PSP to the equivalent new standard.

Providers

Inputs

Resources

Outputs

🚀 Built by opsZero!

<a href="https://opszero.com"><img src="https://opszero.com/wp-content/uploads/2024/07/opsZero_logo_svg.svg" width="300px"/></a>

Since 2016 opsZero has been providing Kubernetes expertise to companies of all sizes on any Cloud. With a focus on AI and Compliance we can say we seen it all whether SOC2, HIPAA, PCI-DSS, ITAR, FedRAMP, CMMC we have you and your customers covered.

We provide support to organizations in the following ways:

• Modernize or Migrate to Kubernetes
• Cloud Infrastructure with Kubernetes on AWS, Azure, Google Cloud, or Bare Metal
• Building AI and Data Pipelines on Kubernetes
• Optimizing Existing Kubernetes Workloads

We do this with a high-touch support model where you:

• Get access to us on Slack, Microsoft Teams or Email
• Get 24/7 coverage of your infrastructure
• Get an accelerated migration to Kubernetes

Please schedule a call if you need support.

<br/><br/>