Awesome
<!-- markdownlint-disable MD033 --> <h1> <img src="https://github.com/sighupio/fury-distribution/blob/main/docs/assets/fury-epta-white.png?raw=true" align="left" width="90" style="margin-right: 15px"/> Kubernetes Fury Ingress </h1> <!-- markdownlint-enable MD033 --><!-- <KFD-DOCS> -->
Kubernetes Fury Ingress provides Ingress Controllers to expose services and TLS certificate management solutions for the Kubernetes Fury Distribution (KFD).
If you are new to KFD please refer to the official documentation on how to get started with KFD.
Overview
Kubernetes Fury Ingress uses CNCF recommended, Cloud Native projects, such as Ingress NGINX an ingress controller using the well-known NGINX server as a URL path-based routing reverse proxy and load balancer, and cert-manager to automate the issuing and renewal of TLS certificates from various issuing sources.
The module also includes additional tools like Forecastle, a web-based global directory of all the services offered by your cluster.
Architecture
The reference architecture used to deploy the Fury Kubernetes Ingress Module is shown in the following figure:
- The traffic from end users arrives first at a Load Balancer that distributes the traffic between the available Ingress Controllers (usually, one for each availability zone).
- Once the traffic reaches the Ingress Controller, the Ingress proxies the traffic to the Kubernetes service based on the URL path of the request.
- The
service
is a Kubernetes abstraction that makes the traffic arrive at the pods where the actual application is running, usually usingiptables
rules.
For more information, please refer to Kubernetes Ingress offiicial documentation
Packages
Kubernetes Fury Ingress provides the following packages:
Package | Version | Description |
---|---|---|
nginx | v1.11.3 | The NGINX Ingress Controller for Kubernetes provides delivery services for Kubernetes applications. |
dual-nginx | v1.11.3 | It deploys two identical NGINX ingress controllers but with two different scopes: public/external and private/internal. |
cert-manager | v1.16.1 | cert-manager is a Kubernetes add-on to automate the management and issuance of TLS certificates from various issuing sources. |
external-dns | v0.15.0 | external-dns allows you to manage DNS records natively from Kubernetes. |
forecastle | v1.0.145 | Forecastle gives you access to a control panel where you can see your ingresses and access them on Kubernetes. |
aws-cert-manager | - | Terraform modules for managing IAM permissions on AWS for cert-manager |
aws-external-dns | - | Terraform modules for managing IAM permissions on AWS for external-dns |
Compatibility
Kubernetes Version | Compatibility | Notes |
---|---|---|
1.28.x | :white_check_mark: | No known issues |
1.29.x | :white_check_mark: | No known issues |
1.30.x | :white_check_mark: | No known issues |
1.31.x | :white_check_mark: | No known issues |
Check the compatibility matrix for additional information on previous releases of the module.
Usage
[!NOTE] Instructions below are for deploying the module using furyctl legacy, that required manual intervention.
Latest versions of furyctl automate the whole process and it is recommended to use the latest version of furyctl instead.
Prerequisites
Tool | Version | Description |
---|---|---|
furyctl | >=0.25.0 | The recommended tool to download and manage KFD modules and their packages. To learn more about furyctl read the official documentation. |
kustomize | >=3.10.0 | Packages are customized using kustomize . To learn how to create your customization layer with kustomize , please refer to the repository. |
Single vs Dual Controller
As the first step, you should choose what type of ingress controller you want to use in your cluster. Kubernetes Fury Ingress provides two types, Single and Dual Ingress Controller.
The Single Controller Package deploys a single class NGINX Ingress Controller that serves all the internal (private) and external (public) traffic.
The Dual Controller Package creates two NGINX Ingress Controller classes, the internal-ingress
and the external-ingress
classes:
-
The
internal-ingress
class is in charge of serving traffic inside the cluster's network, like users accessing via VPN to internal services, for example, application's admin panels or other resources that you don't want to expose to the Internet. -
The
external-ingress
class serves traffic for the applications exposed to the outside of the cluster (e.g., the frontend application to end-users).
Default Configuration
For both Single and Dual NGINX the Kubernetes Fury Ingress module has the following default configuration:
- Maximum allowed size of the client request body:
10m
- HTTP status code used in redirects:
301
- Metrics are scraped by Prometheus every
10s
- Validating Admission webhook that validates an ingress definition does not break NGINX configuration.
Additionally, the following Prometheus Rules for alerts are set up by default:
Parameter | Description | Severity | Interval |
---|---|---|---|
NginxIngressDown | This alert fires if Prometheus target discovery was not able to reach ingress-nginx-metrics in the last 15 minutes. | critical | 15m |
NginxIngressFailureRate | This alert fires if the failure rate (the rate of 5xx responses) measured on a time window of 2 minutes was higher than 10% in the last 10 minutes. | critical | 10m |
NginxIngressFailedReload | This alert fires if the ingress' configuration reload failed in the last 10 minutes. | warning | 10m |
NginxIngressLatencyTooHigh | This alert fires if the ingress 99th percentile latency was more than 5 seconds in the last 10 minutes. | warning | 10m |
NginxIngressLatencyTooHigh | This alert fires if the ingress 99th percentile latency was more than 10 seconds in the last 10 minutes. | critical | 10m |
NginxIngressCertificateExpiration | This alert fires if the certificate for a given host is expiring in less than 7 days. | warning | |
NginxIngressCertificateExpiration | This alert fires if the certificate for a given host is expiring in less than 1 day. | critical |
Deployment
Certificates automation with cert-manager
Kubernetes Fury Ingress cert-manager package is the defacto tool to manage certificates in Kubernetes. It manages the issuing and automatic renewal of certificates for the domains served by the Ingress Controller.
The package's default configuration includes:
- Let's Encrypt as the default certificates issuer, both staging and production.
ClusterIssuer
as the default issuer kind
To deploy the cert-manager
package:
- Add the package to your bases inside the
Furyfile.yml
:
bases:
- name: ingress/dual-nginx
version: "v3.0.1"
- name: ingress/cert-manager
version: "v3.0.1"
-
Execute
furyctl vendor -H
to download the packages -
Inspect the download packages under
./vendor/katalog/ingress/cert-manager
. -
Define a
kustomization.yaml
that includes the./vendor/katalog/ingress/cert-manager
directory as resource.
resources:
- ./vendor/katalog/ingress/cert-manager
For the dual-nginx
you will need to patch the ClusterIssuer
resource with the right ingress class:
---
patchesJson6902:
- target:
group: cert-manager.io
version: v1
kind: ClusterIssuer
name: letsencrypt-staging
path: patches/dual-nginx.yml
- target:
group: cert-manager.io
version: v1
kind: ClusterIssuer
name: letsencrypt-prod
path: patches/dual-nginx.yml
and in the patches/dual-nginx.yml
:
---
- op: "replace"
path: "/spec/acme/solvers/0/http01/ingress/class"
value: "external"
- Finally, execute the following command to deploy the package:
kustomize build . | kubectl apply -f -
NGINX Ingress Controller
⚠️ You'll need to have deployed
cert-manager
first, the validating webhook needs to have TLS communication with the API server.
- Once you selected the type of ingress you want to deploy (
nginx
ordual-nginx
) the next step is to specify this in aFuryfile.yml
:
Single Ingress:
bases:
- name: ingress/nginx
version: "v3.0.1"
Dual Ingress:
dual-nginx
depends on thenginx
package, so we need to download both of them.
bases:
- name: ingress/nginx
version: "v3.0.1"
- name: ingress/dual-nginx
version: "v3.0.1"
See
furyctl
documentation for additional details aboutFuryfile.yml
format.
-
Execute
furyctl legacy vendor -H
to download the packages -
Inspect the download packages under
./vendor/katalog/ingress/
. -
Define a
kustomization.yaml
that includes the./vendor/katalog/ingress
directory as resource.
resources:
- ./vendor/katalog/ingress
-
Apply the necessary patches. You can find a list of common customization here.
-
To deploy the packages to your cluster, execute:
kustomize build . | kubectl apply -f -
Your are now ready to expose your applications using Kubernetes Ingress
objects.
Common customizations
nginx
package is deployed by default as a DaemonSet
, meaning that it will deploy 1 ingress-controller pod on every worker node.
This is probably NOT what you want, standard Fury clusters have at least 1 infra
node (nodes that are dedicated to run Fury infrastructural components, like Prometheus, elasticsearch, and the ingress controllers).
If your cluster has infra
nodes you should patch the daemonset adding the NodeSelector
for the infra
nodes to the Ingress DaemonSet
. You can do this using the following kustomize patch:
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: nginx-ingress-controller-external
spec:
template:
spec:
nodeSelector:
node-kind.sighup.io/infra: ""
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: nginx-ingress-controller-internal
spec:
template:
spec:
nodeSelector:
node-kind.sighup.io/infra: ""
If you don't have infra nodes and you don't want to run ingress-controllers on all your worker nodes, you should probably label some nodes and adjust the previous NodeSelector
accordingly.
Applications directory with Forecastle
Forecastle list all the annotated ingress (applications) that exists in your cluster with an icon grouped by namesapce, in a nice web UI. It lets you search, personalize the header for the landing page (title and colors), it lets you list custom ingress and add more details to each entry.
Use Forecastle as your cluster entry point to discover the running applications easily.
To deploy the forecastle
package:
- Add the package to your bases inside the
Furyfile.yml
:
bases:
- name: ingress/dual-nginx
version: "v3.0.1"
- name: ingress/cert-manager
version: "v3.0.1"
- name: ingress/forecastle
version: "v3.0.1"
-
Execute
furyctl legacy vendor -H
to download the packages -
Inspect the download packages under
./vendor/katalog/ingress/forecastle
. -
Define a
kustomization.yaml
that includes the./vendor/katalog/ingress/forecastle
directory as resource.
resources:
- ./vendor/katalog/ingress/forecastle
- Finally, execute the following command to deploy the package:
kustomize build . | kubectl apply -f -
Basic usage
Forecastle looks for specific annotations on ingresses objects.
Add the following annotations to your ingresses to be discovered by Forecastle:
Annotation | Description | Required |
---|---|---|
forecastle.stakater.com/expose | Add this with value true to the ingress of the app you want to show in Forecastle | true |
forecastle.stakater.com/icon | Icon/Image URL of the application; An icons/logos/images collection repo Icons | false |
forecastle.stakater.com/appName | A custom name for your application. Default is the name of the ingress | false |
forecastle.stakater.com/group | A custom group name. Use if you want the application to show in a different group than the namespace it belongs to | false |
forecastle.stakater.com/instance | A comma-separated list of name/s of the forecastle instance/s where you want this application to appear. Use when you have multiple forecastle dashboards | false |
forecastle.stakater.com/url | A URL for the forecastle app (This will override the ingress URL). It must begin with a scheme i.e. http:// or https:// | false |
forecastle.stakater.com/properties | A comma separate list of key-value pairs for the properties. This will appear as an expandable list for the app | false |
forecastle.stakater.com/network-restricted | Specify whether the application is network restricted or not (true or false) | false |
<!-- Links --> <!-- </KFD-DOCS> --> <!-- <FOOTER> -->See Forecastle official repository for more details.
Contributing
Before contributing, please read first the Contributing Guidelines.
Reporting Issues
In case you experience any problems with the module, please open a new issue.
License
This module is open-source and it's released under the following LICENSE
<!-- </FOOTER> -->