Kubernetes on

Kubernetes CPU Throttled

Sat, 19 Apr 2025 00:00:00 +0000

Kubernetes CPU Requests, Limits, and Throttling

Impacts on Application Performance, some Best Practices

CPU Throttled

CPU Throttling (Kubernetes Workloads): A process where Kubernetes and Container Runtimes restricts a container’s CPU usage to stay within its assigned limits, often causing tasks to wait for the next scheduler quota period when they exceed their allowed CPU time.
CPU Throttling (Linux Kernel): A mechanism in the CFS Bandwidth Control where the kernel prevents a cgroup from exceeding its allocated CPU quota, forcing tasks to stop running until the next quota period.
CPU Requests are implemented via assigning priorities to containers’ processes, i.e., assigning CPU weight and/or “nice” value to cgroups (files “cpu.weight” or “cpu.weight.nice”) Doc: https://docs.kernel.org/admin-guide/cgroup-v2.html#cgroup-v2-cpu Known in community terms also as “Soft Limit”.
CPU Limits are implemented via “CFS Bandwidth Control” (CFS BWC) and cgroups “cpu.max” interface file. CFS BWC Doc: https://docs.kernel.org/scheduler/sched-bwc.html Known in community terms also as “Hard Limit”.

CPU Usage vs CPU Throttling

A random but real example: Grafana CPU Usage vs Throttling reports for a K8s container.

How to explain low CPU Usage, even much below request, but high CPU Throttling? … and this is commonly faced case. What does those Throttling mean? How is Throttling measured?

CPU Management in Kubernetes:

Limits, Requests, Throttling

Requests: Guarantee resources for workloads. In K8s, they affect how Pods are scheduled on nodes. Play role as CPU tasks “weight” (priority) during CPU contention on host machine.
Limits: Cap CPU usage to avoid over-consumption.

K8s CPU and MEM Resources packed onto Node

CPU Throttling: The process of limiting CPU usage when tasks or containers exceed their allocated quota (limit), forcing them to pause and wait until the next scheduling period.

Key Terms and Concepts

CPU: A unit of processing power, which can represent a physical CPU, a CPU core, or a Logical CPU (e.g., a physical core with Hyper-Threading enabled is considered as 2 logical CPUs). In Kubernetes, it’s often referred to as a vCPU, equivalent to a share of processing power.
vCPU (virtual CPU) in cloud and/or virtualized environments – out of scope of this DOC but, in short: is a unit of CPU capacity assigned to a container or pod in Kubernetes. It abstracts the underlying physical hardware, allowing Kubernetes to allocate and manage processing resources in a platform-agnostic way. In most environments, 1 vCPU corresponds to 1 physical CPU or 1 logical CPU (e.g., 1 thread on a hyper-threaded core). On physical hardware, a single physical CPU core with Hyper-Threading enabled may back 2 vCPUs. In virtualized environments (e.g., VMs or cloud instances), multiple vCPUs can be over-committed and mapped to the same physical CPU core. For instance: N:1 mapping: 4 vCPUs could share 1 physical core, with each vCPU getting a fraction of the core’s processing power (e.g., 25% if evenly divided).
CPU Time: The total cumulative time a task or group of tasks spends executing on one or more CPUs. This includes time spent running tasks in parallel (on multiple CPUs) or concurrently (on the same CPU) over a defined period.
CPU Usage: Represents the percentage of CPU capacity consumed by a task or group of tasks over a monitored interval of wall-clock time. It’s a rate (e.g., “50% of one CPU” or “2 CPUs used at 80% i.e. 160%”), typically measured in real-time or averaged over intervals.

CPU Usage and CPU Time are related, they are not identical.

“wall-clock” refers to real-world time as you would measure on a regular clock or stopwatch. It tracks the total elapsed time from start to finish, including everything (e.g., waiting, idle, and active time), not just CPU processing time.
Context switching: the process where the Kernel Scheduler saves the state of a running task (e.g., CPU registers, program counter) and restores the state of another task, allowing the CPU to switch between tasks efficiently. This enables multitasking by sharing CPU time among multiple processes or threads.

Task Types in Relation to CPU Access

CPU-Bound Tasks: Spend most of their time on the CPU, continuously running until their processing is complete. These tasks use the CPU intensively and benefit from more CPU time. The scheduler gives them longer CPU slices, but they are preempted if other tasks have higher priority.
I/O-Bound Tasks: Spend less time on the CPU and frequently yield it to wait for I/O (disk, network). They use short bursts of CPU and quickly return to a waiting state, allowing the scheduler to switch to other tasks.
Memory-Bound Tasks: Performance depends on memory access; they alternate between short CPU bursts and waiting for data from RAM, using the CPU sporadically.
Hybrid Tasks: Some workloads exhibit a mix of characteristics (e.g., web servers handling both CPU-intensive request processing and I/O-heavy database access).

CPU Requests vs K8s Scheduler

Resources (like CPU, MEM) Requests hint K8s Scheduler where (on which node) to bind the pod.

CPU Requests vs Node

Following image may be interpreted as CPU Requests booking on the node: Container A booked 200m CPU or 20% of node Allocatable capacity, Container B - 100m CPU or 10%, For other containers remain 700m CPU or 70% of node Allocatable CPU resource.

Let interpret pie slices like Real CPU Usage: Container A and B use exactly amount of CPU they requested. What if both wants all the node CPU at the same time?

Both containers wants all the node CPU at the same time: Containers will compete for the same CPU but K8s will distribute available CPU to them proportionally by their requests (weight).

Use case 1: A is bursting but B is needing now only 100m CPU, so A can use the whole remaining spare CPU, until…

Use case 2: B is also bursting, so K8s will distribute all available CPU sparely and proportionally to their requests between them…

Other use-case: “borrowing” Container A requested 200m (20% of CPU capacity) but it is using now only 50m (5%). Container B, requested 100m (10%), but it is bursting now and need all CPU, so it may and is using now what is available: 95% It uses 10% of it’s request + 70% of unused and unreserved + 15% borrowed from A which is reserved/requested by A but not used now, so 95%.

CPU Requests vs Limits

Let introduce limits for A and B: A CPU: Requests = 200m, Limits = 260m B CPU: Requests = 100m, Limits = 150m Containers may burst above their requests, but no more then defined limits, beyond which they will be CPU throttled, even if there is a plenty of idle CPU available. Usage slices above requests up to limits are possible only if there are available unused CPU resources on the node.

CPU Limits

CPU limits translate into a time quota for CPU usage within a defined accounting period. CPU Limits = 100m = 0.1 CPU may be translated as: Use 10% of a single CPU capacity/time during a period, or Use as much CPU as needed, but only for 10% of the time within a period. Example: Use the CPU only for 1 second every 10s, or, closer to reality: Use the CPU for 10ms in every 100ms period

CPU Limits vs Throttling

Application has only small (below quota) CPU bursts each period:

Application has bigger CPU bursts each period, and CPU demand exceeds defined quota, so:

CPU Time will be limited to quota Remained time to execute is transferred to next period.

CPU Requests! Always set CPU Requests to realistic and production-like performance testing proven values. They should reflect normal (not minimal) CPU Usage of application, or even p99 of observed maximum, depending by case.

They guarantee requested resources. They allow spare and proportional to requests resources usage when: Is needed higher amount of resources then requested, Resources contention on node,

CPU Limits? This is tricky, controversial subject in community. They may cause application’s big, unexpected and unjustified delays, but they also may… Cap greedy CPU Usage of some buggy crazy noisy neighbor. So…? Set them in function of use case. Setting them requires lots of careful, iterative performance tests! In production set them very high or remove at all for response time critical workloads, especially when there are lots of available CPUs. Set them higher (30-50% to 200-300%) then CPU Request for other cases or when you are enforced to set them.
Set Requests = Limits for narrow use cases, like: Performance Tests to determine workloads sizing and demands When need “Guaranteed QoS” Pod classes Especially Static CPU Binding use cases (narrow usage)

Backups With Velero

Fri, 18 Apr 2025 00:00:00 +0000

What Do You Back Up In Kubernetes?

It’s easy to gloss over backups especially when you are setting up a new cluster, finally getting stuff running on it, and feeling excited that you can now deploy some service and basically automate certificates, ingress, and everything else that used to be a pain to do manually. But as soon as something fails, you will wish you spent a little more time not only thinking about backups, but also documenting step by step how to recover from a disaster.

In Kubernetes, we have two main things to back up: persistent data, and the state of the cluster resources.

With Velero, backups in Kubernetes are EASY. Just do it now, before you put stuff on the cluster, and write down the steps to recover. Why not?

etcd Backups In Talos Linux

Before we dive into Velero, I want to mention that theoretically, you could just use Velero for backups and never worry about etcd. However, it’s a best practice to backup etcd and it’s easy to do, there’s no reason not to have another recovery point.

What is etcd (I think it’s pronounced “et-see-dee”)? This is the database that the control plane nodes use to keep track of any and all resources in the cluster. It’s the cluster database that keeps track of every resource, status, etc. in the cluster. This includes every node, pod, deployment, secret, etc. If you had a catastrophe and lost all your cluster resource state, or had something get corrupted, you should be able to restore the etcd database and get back to the exact state it was in when the backup was taken.

Side note: This tracks real time data so whatever storage the etcd cluster is running on (the actual disks the control plane node stores this data on) needs to be pretty fast. If it’s slow, you can run into issues where kubectl commands are very slow, or stuff stops working as expected in the cluster. Maybe this note belongs in the Talos Linux setup section, but here we are.

Talos Linux, unlike a traditional kubeadm install, does not run etcd as a static pod. Instead, it’s hidden behind the Talos API, meaning you have to use talosctl to manage it: https://www.talos.dev/v1.8/advanced/etcd-maintenance/

How To Back Up etcd In Talos

https://www.talos.dev/v1.8/advanced/disaster-recovery/#backup

talosctl -n etcd snapshot /backup_path/etcd.snapshot.$(/bin/date +%Y%m%d)
- You need to pass any one of the control plane node IPs to the -n flag

You could automate this somewhere so you have regular backups. Highly recommended! If you do a bash script, you’ll need to pass the --talosconfig /path/to/talosconfig option.

If you’re currently in a Disaster Recovery scenario and the snapshot command is failing, copy the etcd database directly: https://www.talos.dev/v1.8/advanced/disaster-recovery/#disaster-database-snapshot

talosctl -n cp /var/lib/etcd/member/snap/db .

How To Recover etcd In Talos

https://www.talos.dev/v1.8/advanced/disaster-recovery/#recovery

I haven’t done this process yet (and hopefully I don’t need it). Follow the official guide.

What’s Velero?

https://velero.io/

Velero is an open source DR and backup/migration/recovery tool for Kubernetes. It backs up all cluster resources (or whatever you specify) and can be used for backup, restore, disaster recovery and migrating to new clusters. It’s also capable of backing up PVs. Backups are stored on external endpoints (mostly S3 capable services including MinIO). Backup/restore is managed through CRDs. There are two components: cluster resources and a client side utility velero.

Don’t trust their blog to be up to date because the current release according to the blog is 1.11, released April 26, 2023, although I’m currently running 1.15 which was released November 5, 2024.

Get an accurate changelog right from the source: https://github.com/vmware-tanzu/velero/releases

Velero Installation/Setup

I’ll walk through installing Velero with some options you might want to consider. Then I’ll dive into running a backup and restore manually, scheduling backups, and backing up persistent volumes with Kopia.

Prerequisites

Access to a Kubernetes cluster, v1.16 or later, with DNS and container networking enabled. For more information on supported Kubernetes versions, see the Velero compatibility matrix.
kubectl installed locally
Object storage. I’m running MinIO in my lab. You could use MinIO, Ceph, AWS S3, Backblaze B2 (S3 API), or any other object storage you prefer.

Installing Velero - Client Side

https://velero.io/docs/v1.15/basic-install/

Install the velero client utility

1
2
3
4
5
6
7


VERSION=v1.15.0
curl -LJO https://github.com/vmware-tanzu/velero/releases/download/$VERSION/velero-$VERSION-linux-amd64.tar.gz
tar zxvf velero-$VERSION-linux-amd64.tar.gz
install -m 755 velero-$VERSION-linux-amd64/velero /usr/local/bin/velero
rm -rf velero-$VERSION-linux-amd64
rm -rf velero-$VERSION-linux-amd64.tar.gz
velero version

Installing Velero - Server Side

There are two installation methods, using the Helm chart or using the velero utility. Let’s use velero CLI since the Helm method seems complicated based on what I can find in the docs.

MinIO Setup

We need to start with an access key for the S3 backup target. I’m using MinIO: https://velero.io/docs/v1.15/contributions/minio/

Create a bucket velero-talos
Create a group velero-svc
Create a service account user velero and add to velero-svc group
Create a policy velero-rw. Raw Policy:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::velero-talos",
                "arn:aws:s3:::velero-talos/*"
            ]
        }
    ]
}

Assign that policy to the velero-svc group
Create an access key on the velero user
Create a file named minio-access-key.txt, replacing the values from your access key

1
2
3


[default]
aws_access_key_id = minio
aws_secret_access_key = minio123

Decide what you think is best for storing this file or the credentials somewhere securely. I would suggest using SOPS, but I will leave the implementation up to you.

Install Using `velero` Utility

Arguments reference:
- --provider - Use the AWS provider which is also used for any generic S3 object storage, including MinIO
- --secret-file - Specifies the secrets to use for authentication against the MinIO storage
- --plugins - Required for Velero to connect to an S3 backend
- --bucket - specifies the name of the bucket on the S3 backend
- --use-volume-snapshots=true - Enables volume snapshots
- --backup-location-config - Configures the connection URL and options for the S3 backend
- --features=EnableCSI - Enables Velero to use a built in CSI snapshot driver, such as democratic-csi and take snapshots using a volumesnapshotclass. To be clear, Velero would create a volumesnapshot and it would be stored wherever democratic-csi stores snapshots (in our case that’s in another dataset on the TrueNAS instance). This is the easy button if you already have CSI snapshots enabled, but the alternative is using Kopia and storing snapshots on the MinIO backend.
Install:
- velero install --provider aws --secret-file minio-access-key.txt --plugins velero/velero-plugin-for-aws:v1.11.0 --bucket velero --use-volume-snapshots=true --backup-location-config region=us-east-1,s3ForcePathStyle="true",s3Url=http://192.168.1.35:9000 --features=EnableCSI
  - You may need to update the plugin version and s3URL.
  - Optionally, set use-volume-snapshots=false if you don’t want to back up PVs.
Verify:
- kubectl get all -n velero
- kubectl -n velero get pod -l deploy=velero
- Check logs, in particular to verify that Velero can reach the backup location
  - kubectl -n velero logs -l deploy=velero
```
level=info msg="BackupStorageLocations is valid, marking as available"
```
Post install:
- If you are using democratic-csi for snapshots, you also need to add a label on the VolumeSnapshotClass to let Velero know which one to use by default. This must only be set on 1 volumeSnapshotClass
  - kubectl edit volumesnapshotclass truenas-iscsi
  1
  
  velero.io/csi-volumesnapshot-class: "true"
- If you’re ADDING this funtionality after you’ve already installed Velero, you just need to modify the server deployment and also enable on the client: velero.io/csi-volumesnapshot-class: “true”
  - Server:
    - kubectl edit -n velero deploy/velero
    1
    
    --features=EnableCSI
  - Client:
    - velero client config set features=EnableCSI
    - velero client config get features

Testing

I’ll go through a specific example that should work to follow along if you’ve followed the series up to now. Otherwise, there are some good examples in the docs that you can reference: https://velero.io/docs/v1.15/examples/

Manual Backup/Restore

Let’s back up the traefik namespace:

velero backup create traefik-backup --include-namespaces traefik
Verify: velero backup describe traefik-backup
- Looking for Phase: Complete
Test a DR scenario:
- Blow away the traefik namespace: kubectl delete ns traefik
- Restore traefik-backup with Velero: velero restore create --from-backup traefik-backup
- Verify: kubectl get ns or velero restore describe traefik-backup-20241130181633 (get the restore name from the previous restore command)

Create a full backup: velero backup create full-20241130 - If you don’t specify namespaces, etc. then everything gets backed up.

Create A Backup Schedule

I like to do daily full backups at 7am. TTL is the expiration time for each backup created, 720h is 30 days. So I get daily backups that fall off after 30 days.

Create schedule: velero schedule create daily-full --schedule "0 7 * * *" --ttl 720h
Verify: velero get schedules

Alternatively, you can use a manifest to define your backup schedule. You may not want to include certain namespaces in your daily backups since most likely you won’t need/want to restore namespaces like kube-system, kube-public, and kube-node-lease. Let’s look at defining a backup schedule using a manifest:

Create the Schedule manifest, schedule.yaml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: daily-skipping-system-namespaces
spec:
  schedule: "0 12 * * *"
  skipImmediately: false
  template:
    excludedNamespaces:
      - kube-public
      - kube-system
      - kube-node-lease

Apply the schedule: kubectl apply -f schedule.yaml
Verify: velero get schedules

Restore From Scheduled Backup

velero restore create --from-schedule SCHEDULE_NAME

Restores In General

See previous examples for specific restore commands. The basic syntax will be velero restore create [RESTORE_NAME] [--from-backup BACKUP_NAME | --from-schedule SCHEDULE_NAME] [flags]

You can restore from a manually created backup or a scheduled backup. You can also restore specific items from a backup, so let’s say you just want to restore your secret key to your sealed secrets from a daily-full scheduled backup, you can use flags to specify exactly what from that backup you want to restore. For example:

velero restore create restore-sealed-secret-key --from-schedule daily-full --selector sealedsecrets.bitnami.com/sealed-secrets-key=active

Back Up PVCs Using democratic-csi Snapshots

If you enabled snapshots in democratic-csi and also enabled snapshos in Velero as described above, then anything you back up with Velero that includes a PVC will be snapshotted. I tested this by deploying a pod/deployment in the default namespace, attached to a test PVC, then ran a Velero backup against it.

velero backup create full-backup

Kopia

Velero uses Kopia (as a plugin) to copy existing snapshots to your S3 backend. This means that if you’re not already creating “regular” snapshots already outside of Velero, you need to have EnableCSI enabled and working already so that a VolumeSnapshot is created, which can then be copied out to S3. In a homelab environment, this may be redundant if your MinIO storage is backed by the same storage you’re using for snapshots, but for the sake of completeness I will run through how we can make this work and test it.

Kubernetes External Services

Fri, 18 Apr 2025 00:00:00 +0000

About External Services

External services are anything you want to route traffic to that does not live in your Kubernetes cluster. For example, you might be running MinIO in a VM and accessing it via IP address, but you want to basically reverse proxy to it. You can use Kubernetes Ingress to act as a reverse proxy to pretty much anything, even if it doesn’t live within your cluster.

It’s pretty straightforward to do this, and you just need to create a few resources: Service, EndpointSlice and Ingress. For the sake of organization, I like to use a separate namespace to separate any external service resources. If you’re doing this, just create a new namespace as your first step.

MinIO Example

I have MinIO running outside my Kubernetes cluster. For now, I just want basically a reverse proxy in front of it, but I want to use Kubernetes Ingress since I already have that running and configured to get TLS certificates from Let’s Encrypt.

Optionally, create a new Namespace. I’m using “externalservices”

1
2
3
4


apiVersion: v1
kind: Namespace
metadata:
  name: externalservices

Ceate an EndpointSlice. This object is the newer replacement for Endpoints which are now managed by EndpointSlice, so it’s no longer recommended to manually create Endpoints directly. EndpointSlice requires a name and a label that matches the service name you will be using. The label’s key is kubernetes.io/service-name. namespace is optional but recommended. Then you just specify address type (IPv4 or IPv6), ports, and the actual endpoint with the IP address. Here’s an example of my EndpointSlice manifest where MinIO listening at 192.168.1.35 on port 80. I’m using a namespace called “externalservices” and the service name is minio-service.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: minio-service
  namespace: externalservices
  labels:
    kubernetes.io/service-name: minio-service
addressType: IPv4
ports:
  - port: 80
endpoints:
  - addresses:
    - "192.168.1.35"
    conditions:
      ready: true

Next, create a ClusterIP Service without a selector. Having no selectors allows the service to be attached directly to an Endpoint that you create instead of only being able to attach to a pod. The way it knows how to attach to the EndpointSlice is based on the label you used in the EndpointSlice manifest which has to exactly match the name you use in this service (in this case minio-service). This service just maps port 80 to targetPort 80, and we will handle HTTP to HTTPS redirection, plus TLS termination in the Ingress resource.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


apiVersion: v1
kind: Service
metadata:
  name: minio-service
  namespace: externalservices
spec:
  type: ClusterIP
  ports:
    - port: 80
      protocol: TCP
      targetPort: 80

Finally, create an Ingress. Here’s a basic example using a subdomain, using a Let’s Encrypt ClusterIssuer, and using traefik as the ingress class.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: minio-service-ingress
  namespace: externalservices
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-staging"
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
spec:
  ingressClassName: traefik
  tls:
  - hosts:
    - minio.example.com
    secretName: tls-example-com
  rules:
  - host: minio.example.com
    http:
      paths:
      - backend:
          service:
            name: minio-service
            port:
              number: 80
        path: /
        pathType: Prefix

TLDR; Put it all together into a single manifest:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58


apiVersion: v1
kind: Namespace
metadata:
  name: externalservices
---
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: minio-service
  namespace: externalservices
  labels:
    kubernetes.io/service-name: minio-service
addressType: IPv4
ports:
  - port: 80
endpoints:
  - addresses:
    - "192.168.1.35"
    conditions:
      ready: true
---
apiVersion: v1
kind: Service
metadata:
  name: minio-service
  namespace: externalservices
spec:
  type: ClusterIP
  ports:
    - port: 80
      protocol: TCP
      targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: minio-service-ingress
  namespace: externalservices
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-staging"
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
spec:
  ingressClassName: traefik
  tls:
  - hosts:
    - minio.example.com
    secretName: tls-example-com
  rules:
  - host: minio.example.com
    http:
      paths:
      - backend:
          service:
            name: minio-service
            port:
              number: 80
        path: /
        pathType: Prefix

Apply kubectl apply -f minio-external-service.yaml
Shortly after deploying the resources, update DNS or your local hosts file and point to minio.example.com. It may take a bit to generate a signed certificate from the issuer, but you should still be able to route to your endpoint through the Ingress and verify that it’s working.

That’s it. That’s the end. You are now done.

Kubernetes: Certificates With cert-manager and Let's Encrypt

Fri, 18 Apr 2025 00:00:00 +0000

What is cert-manager?

https://cert-manager.io/
cert-manager is a certificate controller for Kubernetes which is capable of handling all your certificate needs. This of course includes acquiring and automatically renewing certificates from Let’s Encrypt, but it can also be used as a local CA (certificate authority) for private certificates between services, etc.

Installation/Setup

https://cert-manager.io/docs/
I run piHole internally on my network and also use DNS challenge for internal only hostnames. This means when I request a new certificate and cert-manager attempts to look up the DNS challenge record, it won’t be able to query it through my piHole.

The solution is to configure cert-manager to specifically use public DNS servers to do the lookup, and that is done by setting dns01-recursive-nameservers and dns01-recursive-nameservers-only.

For that reason, I don’t use the “Default static install” method by just installing the manifest using kubectl. Instead, I use the Helm chart so that I can apply those overrides during the installation.

If you don’t need to override the nameservers used for DNS challenge, I would recommend using their Default static install method: https://cert-manager.io/docs/installation/#default-static-install

If you’re still with me and want to apply using the Helm chart, follow along!

Make sure you have Helm version 3 or later: https://helm.sh/docs/intro/install/

Add the helm repo:

1

helm repo add jetstack https://charts.jetstack.io --force-update

Install the Helm chart. Note the extraArgs for DNS nameserver settings:

1
2
3
4
5
6
7


helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.16.1 \
  --set crds.enabled=true
  --set 'extraArgs={--dns01-recursive-nameservers-only,--dns01-recursive-nameservers=8.8.8.8:53\,1.1.1.1:53}'

If you’re doing anything fancy on your network like me, double check that your Kubernetes nodes running cert-manager are able to reach the dns01-recursive-nameservers on port 53

Verify deployment: kubectl get all -n cert-manager
(optional) Verify dns01-recursive-nameserver settings were applied: kubectl -n cert-manager describe deploy cert-manager | grep dns01

Verify Installation

At this point cert-manager should be running and ready to issue self-signed certificates. We can test that.

Test with this file named cert-manager-verify.yaml:

Apply: kubectl apply -f cert-manager-verify.yaml
Verify: kubectl describe cert -n cert-manager-test
Cleanup: kubectl delete -f cert-manager-verify.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


apiVersion: v1
kind: Namespace
metadata:
  name: cert-manager-test
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: test-selfsigned
  namespace: cert-manager-test
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: selfsigned-cert
  namespace: cert-manager-test
spec:
  dnsNames:
    - example.com
  secretName: selfsigned-cert-tls
  issuerRef:
    name: test-selfsigned

Getting Certificates From Let’s Encrypt (using Cloudflare DNS)

Now we are ready to hook up Let’s Encrypt so we can get certificates that are signed by a trusted authority. It’s always a good idea to start with the staging issuer when testing Let’s Encrypt so you don’t run into rate limits with their production infrastructure.

I’m using Cloudflare for my DNS nameservers so these steps will be based on that setup.

Getting A Cloudflare API Token

Create a Kubernetes secret resource with your Cloudflare API token. This allows cert-manager to add custom _acme-challenge records for domain validation. This will be the same API token you use for both Staging and Production certificates, so you only need to create one of these.
- In Cloudflare, you can generate a new API token by navigating to your user account > My Profile > API Tokens > Create Token. Make sure token permissions include All zones - Zone Settings:Read, Zone:Read, DNS:Edit or you can limit to a specific zone if you have multiple and want to use different API Tokens for different zones.
1 2 3 4 5 6 7 8

apiVersion: v1 kind: Secret metadata: name: letsencrypt-cloudflare-api-token-secret namespace: cert-manager type: Opaque stringData: api-token:
Apply the secret manifest: kubectl apply -f letsencrypt-cloudflare-api-token-secret.yaml

Deploying A cert-manager Issuer - Staging

cert-manager has two types of issuers: Issuer and ClusterIssuer. Issuer is used for more fine-grained control, it can be placed within a specific namespace, etc. ClusterIssuer, as the name implies, can be accessed across the whole Kubernetes cluster. Since I’m not getting too complicated, and also want to be able to easily utilize cert-manager from multiple namespaces, I’m using ClusterIssuer. If you wanted to use Issuer, just pick a namespace and the steps are almost identical to this.

It’s strongly recommended to test with Let’s Encrypt’s staging server to avoid any rate limiting. You are far less likely to run into rate limits during testing while using the staging issuer, and once you get this working it’s simply a matter of updating the ACME server URL to the production issuer and changing the name on the Issuer/ClusterIssuer (or you can run boht at the same time).

https://letsencrypt.org/docs/staging-environment/

You can use any valid email address. When you request a new cert, the email address is registered with that particular request, and that’s where they will send renewal/expiry notices.

Create a manifest file for the staging issuer letsencrypt-staging-clusterissuer.yaml. Update your email address:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    # The ACME server URL
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: mail@example.com
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-staging-issuer-secret
    # Enable the DNS-01 challenge provider
    solvers:
    - dns01:
        cloudflare:
          apiTokenSecretRef:
            name: letsencrypt-cloudflare-api-token-secret
            key: api-token

Apply the staging issuer: kubectl apply -f letsencrypt-staging-clusterissuer.yaml
Verify your ClusterIssuer exists (or Issuer if that’s what you’re using): kubectl get clusterissuer

Request A Real (Staging) Certificate From Let’s Encrypt

Now we need to deploy a Certificate resource (again) to test our Let’s Encrypt staging ClusterIssuer. This is similar to the verify manifest like before, but only needs to include the Certificate resource. However, we also need to update a couple things, notably adding the field Certificate.spec.issuerRef.kind and setting its value to ClusterIssuer (see https://cert-manager.io/docs/usage/certificate/)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: letsencrypt-staging-certificate
  namespace: default
spec:
  dnsNames:
    - justkidding.example.com
  secretName: letsencrypt-staging-cert-tls
  issuerRef:
    kind: ClusterIssuer
    name: letsencrypt-staging

Apply the Certificate manifest: kubectl apply -f letsencrypt-staging-certificate.yaml
Check the Certificate is ready: kubectl get certificate - this will show Ready=False until it’s fully issued
Be patient. This can sometimes be quick, but sometimes can take a while. My most recent attempt took 38 minutes (see Troubleshooting section below to dive into what’s happening during the process).

Once you get a certificate issued using the staging issuer, you are ready to move to production!

Deploying A cert-manager Issuer - Production

Follow the same steps as in Staging, but using the production ACME URL and probably not using the name “staging”.

Create a manifest file for the production issuer letsencrypt-production-clusterissuer.yaml. Update your email address:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-production
spec:
  acme:
    # The ACME server URL
    server: https://acme-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: mail@example.com
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-production-issuer-secret
    # Enable the DNS-01 challenge provider
    solvers:
    - dns01:
        cloudflare:
          apiTokenSecretRef:
            name: letsencrypt-cloudflare-api-token-secret
            key: api-token

Apply the staging issuer: kubectl apply -f letsencrypt-production-clusterissuer.yaml
Verify your ClusterIssuer exists (or Issuer if that’s what you’re using): kubectl get clusterissuer

Request A Real (Production) Certificate From Let’s Encrypt

This is exactly the same as for staging, except we make the request from the production ClusterIssuer (the last line in the yaml file says which ClusterIssuer to use).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: letsencrypt-production-certificate
  namespace: default
spec:
  dnsNames:
    - justkidding.example.com
  secretName: letsencrypt-production-cert-tls
  issuerRef:
    kind: ClusterIssuer
    name: letsencrypt-production

Apply the Certificate manifest: kubectl apply -f letsencrypt-production-certificate.yaml
Check the Certificate is ready: kubectl get certificate - this will show Ready=False until it’s fully issued
Be patient. This can sometimes be quick, but sometimes can take a while depending on a lot of factors. See Troubleshooting below for some tips.

Troubleshooting Certificate Requests

Definitive guide: https://cert-manager.io/docs/troubleshooting/
- Also for Let’s Encrypt: https://cert-manager.io/docs/troubleshooting/acme/
kubectl get certificaterequests
kubectl describe certificaterequests
kubectl get order
kubectl describe order
kubectl get challenge
kubectl describe challenge
- This level will tell you if it’s stuck waiting for DNS challenge (basically it’s waiting for the _acme-challenge record to be added and validated from the cert-manager service. This is where sometimes you can get timeouts if your network doesn’t allow cert-manager to query public DNS servers to check).
- Troubleshooting steps I’ve had here were basically verifying that the DNS record exists in Cloudflare while it’s waiting. If it is, wait a while to see if it eventually solves the challenge. If not, you might need to dive deeper into dns01-recursive-nameserver settings (see above) or otherwise make sure your internal DNS resolver is able to query the new record.
- In the output, look for Challenge.Spec.Key which is the value you should see in the _acme-challenge TXT record
You can also check the logs on the cert-manager container
- Find the pod name: kubectl get po -n cert-manager
- kubectl logs cert-manager-556766675f-pt123 -n cert-manager
See cert-manager issue for more discussion: https://github.com/cert-manager/cert-manager/issues/5917

What About Ingress?

I’ll revisit this topic after we get to Ingress Controllers (using Traefik) and how to get certificates from Let’s Encrypt. It’s significantly easier than this, and since we already have this production ClusterIssuer in place, you basically just add a line in your Ingress resource to use this ClusterIssuer to attach a certificate and cert-manager does the rest!

OpenEBS Replicated Storage Mayastor

Fri, 18 Apr 2025 00:00:00 +0000

Intro and Prerequisites

In a previous post, I mentioned that I struggled to get OpenEBS working in Talos and instead went with democratic-csi. In recent weeks, I decided to revisit this and figure out how to get OpenEBS replicated storage working in order to evaluate replicated storage in my cluster. I now have multiple disks that I can dedicate to my Kubernetes cluster and wanted to avoid the issue with the single point of failure using a TrueNAS VM for democratic-csi.

If you are following along, I will assume you are familiar with deploying Talos Linux itself and have talosctl installed with an existing cluster running. If you need more details on how to do that, check out https://blog.dalydays.com/post/kubernetes-homelab-series-part-1-talos-linux-proxmox/.

Dedicated Storage Node

It’s not absolutely necessary to use a dedicated storage node. I’m doing this because I want to pass a disk directly to a VM for storage on each physical host and want to keep storage somewhat isolated from other worker nodes, and I can spare the few extra resources to dedicate to this purpose. If you want to use existing worker nodes, just follow this process for your existing nodes instead of creating new ones.

Create New Talos Node(s)

Create a VM in Proxmox with 4GB RAM and 4 vCPU cores (2GB RAM is not enough due to the fact that you will be enabling hugepages which takes up 2GB and you would see oom-kills otherwise. You also need a dedicated CPU core just for the io-engine, along with all the other stuff that runs. I tried with 2vCPU and it wouldn’t schedule the io-engine pod due to insufficient resources). I named my first one talos-storage-1
- Attach a Talos ISO to the CD ROM and boot from it
- Get the IP address from the node
Install Talos using the worker.yaml template used for other worker nodes (you may want to get a current or updated version of Talos from the image factory):
- talosctl apply-config --insecure -n 10.0.50.135 --file _out/worker.yaml
Apply a patch to set a static IP and node label, e.g.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


# ./patches/storage1.patch
machine:
  network:
    hostname: talos-storage-1
    interfaces:
      - deviceSelector:
          busPath: "0*"
        addresses:
          - 10.0.50.31/24
        routes:
          - network: 0.0.0.0/0
            gateway: 10.0.50.1
    nameservers:
      - 192.168.1.22

talosctl patch mc -n 10.0.50.135 --patch @patches/storage1.patch
- I’m having trouble here with the node name changing, and I have to manually delete the random name from the cluster:
- e.g. kubectl delete node talos-lry-si8
- Also you might need to delete the label openebs.io/nodename= if you already have openebs running and are adding/changing nodes
  - kubectl edit node talos-storage-1 and change the value to the current node name
Apply a patch to set some machine config stuff for OpenEBS which includes hugepages, a nodeLabel for where mayastor engine should run, and the /var/local bind mount:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


# ./patches/openebs.patch
machine:
  sysctls:
    vm.nr_hugepages: "1024"
  nodeLabels:
    openebs.io/engine: mayastor
  extraMounts:
    - destination: /var/local
      type: bind
      source: /var/local
      options:
        - rbind
        - rshared
        - rw

talosctl patch mc -n 10.0.50.31 --patch @patches/openebs.patch
If you have an additional disk to use with OpenEBS, you’ll need to pass it directly to the Talos node VM. I’m using Proxmox
- SSH into the Proxmox host and find the disk ID to be passed. I just run ls -lh /dev/disk/by-id/ and get the root disk (not containing any "_1" or "_part*", for example /dev/disk/by-id/nvme-Samsung_SSD_970_EVO_Plus_2TB_S59CNM0W635077P`)
- Pass the disk directly to the Talos VM, where 511 is the VM ID and assuming you only have 1 disk already on scsi0: qm set 511 -scsi1 /dev/disk/by-id/nvme-Samsung_SSD_970_EVO_Plus_2TB_S59CNM0W635077P
- Checking the hardware tab on VM 511 in Proxmox, you should see this new disk. Double click it and make sure to check “Advanced”, “Discard”, and “SSD emulation”
  - If you see orange on these settings, you will need to shut down the VM, then power it back on for the changes to apply. Rebooting won’t do it.
- Now that the disk has been added, look for it with talosctl: talosctl get disks -n 10.0.50.31
  - In my case I see a disk named sdb which is 2.0TB with model “QEMU HARDDISK”
- Mount the disk to be passed to containers with appropriate privileges. This is required for openebs-io-engine to access the extra disk.
1 2 3 4

# ./patches/mount-sdb.patch machine: disks: - device: /dev/sdb
- Apply: talosctl patch mc -n 10.0.50.31 --patch @patches/mount-sdb.patch - At this point the Talos node will reboot and should come back up healthy in a minute.
- View the console or check the dashboard with talosctl dashbord -n 10.0.50.31
- If you see an error about being unable to mount the disk or the partition being the wrong type, etc. you will need to wipe the disk and create a fresh GPT partition. As of Talos 1.9.0 this can be done with talosctl wipe sdb -n 10.0.50.31, otherwise you would need to do this outside of Talos.
  - talosctl wipe disk sdb -n 10.0.50.31 - where sdb is the device, you confirmed this right? Confirm using talosctl get disks -n 10.0.50.31
  - Otherwise from Proxmox you could do this: Shut down the VM and do this in proxmox with wipefs /dev/yourdev and then use fdisk /dev/yourdev > g > w (g writes a new GPT table, w writes to disk). Now power Talos back on and it should do its thing.
  - Yet another option would be to boot into a different Linux ISO on the VM and use a tool like Gparted. Whatever you like best.

Let’s Verify Our Disk Mount

When Talos successfully mounts the extra disk, we should see it listed with lsblk without any partitions. We want to pass the raw disk to OpenEBS. In order to check, run a debug pod on your storage node and check the bind mounts.

kubectl debug node/talos-storage-1 -it --image=alpine -- /bin/sh
apk add lsblk
lsblk
Check for the mount, showing the full capacity of your disk.

Now repeat this whole process for any other Talos nodes you need. I have 3, so I’m doing talos-storage-1, talos-storage-2 and talos-storage-3.

Worker Nodes Also Need /var/local Mounted

Certain OpenEBS components run on any node, and this requires all worker nodes to have /var/local mounted.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# ./patches/mount-var-local.patch
machine:
  kubelet:
    extraMounts:
      - destination: /var/local
        type: bind
        source: /var/local
        options:
          - rbind
          - rshared
          - rw

In my case, I applied this to my 3 worker nodes. I don’t think a reboot is required, but you could if you wanted to:

talosctl patch mc -n 10.0.50.21 --patch @patches/mount-var-local.patch
talosctl patch mc -n 10.0.50.22 --patch @patches/mount-var-local.patch
talosctl patch mc -n 10.0.50.23 --patch @patches/mount-var-local.patch

In order to check the other bind mount for /var/local, we have to wait until after deploying OpenEBS because the mount isn’t utilized until a pod is deployed with a HostPath volume at or below this path. Specifically, the openebs-io-engine-* Daemonset maps to this path.

Installing OpenEBS

This was a pain to figure out. Documentation from OpenEBS is lacking, and so is documentation from Talos on the same topic. Here’s what I found to work. You need a privileged namespace, bind mounts on all worker nodes, then DiskPools before you can start testing PVCs.

Privileged Namespace

OpenEBS requires privileges, and the easiest way to handle that is by making the namespace privileged (rather than messing with machine configs).

Add a new privileged namespace. The Helm chart wants you to use openebs so do this:

1
2
3
4
5
6
7
8
9


# namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: openebs
  labels:
    pod-security.kubernetes.io/enforce: privileged
    pod-security.kubernetes.io/warn: privileged
    pod-security.kubernetes.io/audit: privileged

kubectl apply -f namespace.yaml

Helm Installation

helm repo add openebs https://openebs.github.io/openebs
helm repo update
Grab the values from the Helm chart (helm show values openebs/openebs > values.yaml), or use this. I have already modified the config to disable initContainers which is a known issue with Talos, and disabled local provisioners that I’m not interested in.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65


# values.yaml
openebs-crds:
  csi:
    volumeSnapshots:
      enabled: true
      keep: true

# Refer to https://github.com/openebs/dynamic-localpv-provisioner/blob/v4.1.2/deploy/helm/charts/values.yaml for complete set of values.
localpv-provisioner:
  rbac:
    create: true

# Refer to https://github.com/openebs/zfs-localpv/blob/v2.6.2/deploy/helm/charts/values.yaml for complete set of values.
zfs-localpv:
  crds:
    zfsLocalPv:
      enabled: true
    csi:
      volumeSnapshots:
        enabled: false

# Refer to https://github.com/openebs/lvm-localpv/blob/lvm-localpv-1.6.2/deploy/helm/charts/values.yaml for complete set of values.
lvm-localpv:
  crds:
    lvmLocalPv:
      enabled: true
    csi:
      volumeSnapshots:
        enabled: false

# Refer to https://github.com/openebs/mayastor-extensions/blob/v2.7.2/chart/values.yaml for complete set of values.
mayastor:
  csi:
    node:
      initContainers:
        enabled: false
  etcd:
    # -- Kubernetes Cluster Domain
    clusterDomain: cluster.local
  localpv-provisioner:
    enabled: false
  crds:
    enabled: false

# -- Configuration options for pre-upgrade helm hook job.
preUpgradeHook:
  image:
    # -- The container image registry URL for the hook job
    registry: docker.io
    # -- The container repository for the hook job
    repo: bitnami/kubectl
    # -- The container image tag for the hook job
    tag: "1.25.15"
    # -- The imagePullPolicy for the container
    pullPolicy: IfNotPresent

engines:
  local:
    lvm:
      enabled: false
    zfs:
      enabled: false
  replicated:
    mayastor:
      enabled: true

helm install openebs -n openebs openebs/openebs -f values.yaml
Verify: kubectl get po -n openebs and it should look something like this:

NAME                                          READY   STATUS    RESTARTS      AGE
openebs-agent-core-74d4ddc7c5-hjnxl           2/2     Running   0             9m23s
openebs-agent-ha-node-f9bsb                   1/1     Running   0             9m23s
openebs-agent-ha-node-gjdbt                   1/1     Running   0             9m23s
openebs-agent-ha-node-mwjq9                   1/1     Running   0             9m23s
openebs-agent-ha-node-rfjrw                   1/1     Running   0             93s
openebs-api-rest-757d87d4bd-zd2ms             1/1     Running   0             9m23s
openebs-csi-controller-58c7dfcd5b-6jtcq       6/6     Running   0             9m23s
openebs-csi-node-fmmwg                        2/2     Running   0             9m23s
openebs-csi-node-j95f5                        2/2     Running   2 (60s ago)   93s
openebs-csi-node-jxkvq                        2/2     Running   0             9m23s
openebs-csi-node-xtsnt                        2/2     Running   0             9m23s
openebs-etcd-0                                1/1     Running   0             9m23s
openebs-etcd-1                                1/1     Running   0             9m23s
openebs-etcd-2                                1/1     Running   0             9m23s
openebs-io-engine-lb8zr                       2/2     Running   0             9m23s
openebs-localpv-provisioner-657c44878-wjmwr   1/1     Running   0             9m23s
openebs-loki-0                                1/1     Running   0             9m23s
openebs-nats-0                                3/3     Running   0             9m23s
openebs-nats-1                                3/3     Running   0             9m23s
openebs-nats-2                                3/3     Running   0             9m23s
openebs-obs-callhome-8665bb8f6f-4ntrd         2/2     Running   0             9m23s
openebs-operator-diskpool-6d44884f8f-52rrx    1/1     Running   0             9m23s
openebs-promtail-2g6kz                        1/1     Running   0             9m23s
openebs-promtail-d72cx                        1/1     Running   0             93s
openebs-promtail-hsxlc                        1/1     Running   0             9m23s
openebs-promtail-npw7f                        1/1     Running   0             9m23s

If pods are stuck initializing after a few minutes, start checking logs with openebs-etcd-0 since a lot of things depend on that being up before it will initialize.
- Related to /var/local bind mounts not being added to worker nodes, openebs-etcd-* will have Warning events about “FailedMount”, stating that a PVC doesn’t exist. If you look closely, the path starts with /var/local/... which means you’re missing the /var/local bind mount on one or more Talos worker/storage nodes.
- The fix in this situation would be to fix the bind mount on the worker nodes, and you could also try a reboot (you can identify the node based on the pod having issues). There’s no need to change the OpenEBS deployment.
- After fixing it, you might see stale pods with errors. You can terminate pods in an error state, and if they are needed the Deployment or Daemonset responsible for them will recreate them as needed.

Add DiskPool(s)

I was excited at this point to test with a PVC, but then was confused about why it wouldn’t provision anything. The Talos docs feel a bit sparse and seem to imply that now is the time to test with a PVC. But if you pay close attention they only mention testing the local provisioner, adding to my confusion. It turns out you need to add DiskPools, which makes sense in hindsight. If you’ve ever used Longhorn there is a similar config needed after the initial install, so that you know what disk capacity you have to work with.

Earlier, we mounted that 2TB disk in the talos-storage-1 node. Now we’ll use that for our first DiskPool
Get the disk ID by exec-ing into the openebs-io-engine pod
- Identify one of the io-engine pods: kubectl get po -l app=io-engine -n openebs
- Exec into the pod: kubectl exec -it openebs-io-engine-jpnrh -c io-engine -n openebs -- bin/sh
- ls -lh /dev/disk/by-id/ - grab the one pointing to /dev/sdb in our case, which for me is scsi-0QEMU_QEMU_HARDDISK_drive-scsi1
FYI I went with uring instead of aio since it’s the new kid on the block

1
2
3
4
5
6
7
8
9


# diskpool-1.yaml
apiVersion: "openebs.io/v1beta2"
kind: DiskPool
metadata:
  name: pool-1
  namespace: openebs
spec:
  node: talos-storage-1
  disks: ["uring:///dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1"]

kubectl apply -f diskpool-1.yaml
Verify: kubectl get dsp -n openebs - quickly it should change to Created state and Online POOL_STATUS

Repeat this process for any/all storage nodes you have. Since I’m virtualizing Talos, the disk path is exactly the same on all 3 nodes so I can reuse the config, just updating the pool name and the node name.

Troubleshooting

Sorry I can’t be a ton of help here since I have only done really limited troubleshooting. If you run into issues with diskpools stuck in Creating status, you will need to describe the dsp or check logs.

What I can say is that I’m running a homelab, which means I have 3 different disks, from 3 different manufacturers. Two of them worked in this configuration, but one did not. The disk is fine, it’s fairly new, I’ve tried wiping it multiple times, multiple ways, but it just would not work with the diskpool. I tried doing it as a bind mount and using /mnt/local/nvme2tb (this one requires adding the volume to the io-engine Daemonset), mounting /dev/sdb, mounting /dev/sdb1, everything I could think of, but it would not create. I rebuilt the Talos node but got the same results. I switched to a different disk and changed nothing else, and it works totally fine. For prosperity, these are the disks I have and which ones worked in this configuration.

Samsung 970 EVO Plus 2TB - no problems
Samsung 990 EVO 2TB - no problems
WD BLACK SN770 2TB - no problems
Crucial P3 Plus 2TB (CT2000P3PSSD8) - COULD NOT GET THIS WORKING :(

If you are reading this and you know or think you might know why this didn’t work, please reach out! I’m interested in understanding why this wouldn’t work and how I could troubleshoot better.

Testing A Replicated PVC

If you are here, you have at least one working diskpool and are ready to test that PVC provisioning works and can be attached to a running pod. Let’s test that.

Verify diskpools: kubectl get dsp -n openebs - for my 3 storage nodes with 2TB volumes, I’m seeing this

1
2
3
4


NAME     NODE              STATE     POOL_STATUS   CAPACITY        USED   AVAILABLE
pool-1   talos-storage-1   Created   Online        1998443249664   0      1998443249664
pool-2   talos-storage-2   Created   Online        1998443249664   0      1998443249664
pool-3   talos-storage-3   Created   Online        1998443249664   0      1998443249664

Check what storage classes are available: kubectl get sc

1
2
3
4
5


NAME                     PROVISIONER               RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
mayastor-etcd-localpv    openebs.io/local          Delete          WaitForFirstConsumer   false                  6h15m
mayastor-loki-localpv    openebs.io/local          Delete          WaitForFirstConsumer   false                  6h15m
openebs-hostpath         openebs.io/local          Delete          WaitForFirstConsumer   false                  6h15m
openebs-single-replica   io.openebs.csi-mayastor   Delete          Immediate              true                   6h15m

For now, we’re interested in testing that openebs-single-replica SC that uses Mayastor, so write this file. Note that this uses the default namespace:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


# test-pvc-openebs-single-replica.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: openebs-testpvc
spec:
  storageClassName: openebs-single-replica
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

Apply it: kubectl apply -f test-pvc-openebs-single-replica.yaml
Check PV and PVC:
- kubectl get pv
- kubectl get pvc - you should see a PVC named openebs-testpvc with status Bound and storage class openebs-single-replica
Deploy a test pod to attach the PVC to - this pod is also in the default namespace:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


# pod-using-testpvc.yaml
---
apiVersion: v1
kind: Pod
metadata:
  name: testlogger
spec:
    containers:
    - name: testlogger
      image: alpine:3.20
      command: ["/bin/ash"]
      args: ["-c", "while true; do echo \"$(date) - test log\" >> /mnt/test.log && sleep 1; done"]
      volumeMounts:
      - name: testvol
        mountPath: /mnt
    volumes:
    - name: testvol
      persistentVolumeClaim:
        claimName: openebs-testpvc

kubectl apply -f pod-using-testpvc.yaml
Verify it’s running: kubectl get po testlogger
Exec into the test pod: kubectl exec -it testlogger -- /bin/sh
Look at your mounts with df -h /mnt. Since OpenEBS Mayastor uses NVMe-oF you should see that mount path /mnt attached to what looks like a NVMe block device.

1
2


Filesystem                Size      Used Available Use% Mounted on
/dev/nvme0n1              9.7G     28.0K      9.2G   0% /mnt

Hmm. The size doesn’t quite add up. 9.7G is close to 10Gi but then 28.0K used does not equal 9.2G available. Hmm…
Cleanup:
- kubectl delete -f pod-using-testpvc.yaml
- kubectl delete -f test-pvc-openebs-single-replica.yaml

What Is A “Single” Replica Anyway?

A replica is an exact reproduction of something…

A replica by definition cannot exist without copying something that already exists, which inherently means there must be at least 2. In the context of OpenEBS, a single replica just means you only have ONE copy of the data. It gets randomly assigned to one of the diskpools available. If that disk fails, that data is gone. But we are using OpenEBS for the purpose of replication, so how do we get more replicas??? Follow me.

Create a new storage class with 2 replicas (feel free to do 3 or any value at this point):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# openebs-2-replicas-sc.yaml
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: openebs-2-replicas
parameters:
  protocol: nvmf
  repl: "2"
provisioner: io.openebs.csi-mayastor

kubectl apply -f openebs-2-replicas-sc.yaml
Verify: kubectl get sc
Test - deploy another test-pvc using the new SC:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


# test-pvc-openebs-2-replicas.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: openebs-testpvc
spec:
  storageClassName: openebs-2-replicas
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

kubectl apply -f openebs-2-replicas.yaml
Test with a pod, using the same pod from earlier - kubectl apply -f pod-using-testpvc.yaml
Exec into the test pod: kubectl exec -it testlogger -- /bin/sh
Check your mounted disk: df -h /mnt

Conclusion

I’m going to stop here. I didn’t go into any details about how to check which diskpools hold the replica(s) for the PV, but I assume there is a way to do that. I also did not look at how to recover in case there is a diskpool or storage node failure. Assuming you had 3 replicas, and one went down, there would be no data loss.

I didn’t cover performance, monitoring, recovery, or anything else that you probably care about long term. That could be a future post, but my next stop is actually evaluating Longhorn with the v2 engine. As of today, they have released 1.8.0-rc5, which enables support for their v2 engine with Talos (this just means they support NVMe-oF). If Longhorn can now work with NVMe-oF and Talos, to me that is a much more mature and feature rich product, with more community support than OpenEBS. I believe it also supports snapshots and other features that OpenEBS does not currently.

My next post will be all about blowing this setup away and doing it all over with Longhorn. Hopefully by then the stable 1.8.0 will have been released.

Keycloak as an OIDC provider for Kubernetes

Thu, 17 Apr 2025 00:00:00 +0000

Keycloak as an OIDC provider for Kubernetes

The workflow

The client requests an ID Token with claims for it’s identity (name) and the groups he/she belongs to
The client then requests access to Kubernetes providing the ID token from the IDP obtained previously
This token (which contains the claims for name, group ) is used in each request to the API Server
The API Server in turn checks the ID Token validity with the ID provider
If the token is valid, then the API Server will check if the request is authorized based on the token’s claims and the configured RBAC (by matching it with the corresponding resources)
Finally, the actions will be performed or denied
A response is sent back to the client

From the user perspective, once everything is setup, we will perform this actions to obtain access to the cluster:

Get an ID Token (and Refresh token) from the ID provider (we will request the tokens from Keycloak).
Set a user’s credentials for kubectl .
Set a new kubectl config with this user and a configured cluster (for example, minikube )
Done: Use the config, issue commands

From the Keycloak admin’s perspective, we will:

Create a client (in our example, a public client, i.e.: no client secret)
Create some basic claims for identification and management of users and groups, specifically:
- name
- groups
Place the target users within the corresponding group

From the Kubernetes admin’s perspective, we will:

Configure the required RBAC resources: for example, a ClusterRole with the permitted operations and a ClusterRoleBinding that matches the desired group.
Configure the API Server to use Keycloak as an OIDC provider

Configure a Keycloak client for Kubernetes SSO

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57


{
  "clientId": "k8s",
  "name": "k8s",
  "description": "Kubernetes SSO",
  "rootUrl": "",
  "adminUrl": "",
  "baseUrl": "",
  "surrogateAuthRequired": false,
  "enabled": true,
  "alwaysDisplayInConsole": false,
  "clientAuthenticatorType": "client-secret",
  "redirectUris": [
    "/*"
  ],
  "webOrigins": [
    "/*"
  ],
  "notBefore": 0,
  "bearerOnly": false,
  "consentRequired": false,
  "standardFlowEnabled": true,
  "implicitFlowEnabled": false,
  "directAccessGrantsEnabled": true,
  "serviceAccountsEnabled": false,
  "publicClient": true,
  "frontchannelLogout": true,
  "protocol": "openid-connect",
  "attributes": {
    "oidc.ciba.grant.enabled": "false",
    "oauth2.device.authorization.grant.enabled": "false",
    "backchannel.logout.session.required": "true",
    "backchannel.logout.revoke.offline.tokens": "false"
  },
  "authenticationFlowBindingOverrides": {},
  "fullScopeAllowed": true,
  "nodeReRegistrationTimeout": -1,
  "defaultClientScopes": [
    "web-origins",
    "acr",
    "profile",
    "roles",
    "name",
    "groups",
    "email"
  ],
  "optionalClientScopes": [
    "address",
    "phone",
    "offline_access",
    "microprofile-jwt"
  ],
  "access": {
    "view": true,
    "configure": true,
    "manage": true
  }
}

I am allowing all redirects and all web origins, though this is less than desirable in production.
Please change this values to your redirect URLs to enhance the security.

I have let only the Standard Flow and the Direct access (for username and password sing-in).

Configuring the name and groups claims

We want to make those two claims available in the ID token. For that we will:

Create new client scopes and
Specify how to map them to the token
Configure those new scopes in the Keycloak client that we created for Kubernetes authentication

The group scope:

Gotchas

Forgetting to strip the path from the groups’s name
Forgetting to add this to the ID Token!
Forgetting to add this to the user info (if you plan to validate the token before submitting it in each request)

The “Kubernetes” Client, Client Scopes configuration:

Add the client scopes name and groups
Make them Default so that’s easier later on

Common CLI environment variables

To make it easier to go through the rest of the steps we will use some environment variables:

1
2
3
4
5
6


export REALM='the_realm_that_contains_the_k8s_client'
export OIDC_SERVER='https://your.keycloak.server.local:8443'
export OIDC_ISSUER_URL="${OIDC_SERVER}/realms/${REALM}"
export OIDC_CLIENT_ID=k8s
export OIDC_TOKEN_ENDPOINT=$(curl "${OIDC_ISSUER_URL}/.well-known/openid-configuration" | jq -r '.token_endpoint')
export OIDC_USERINFO_ENDPOINT=$(curl "${OIDC_ISSUER_URL}/.well-known/openid-configuration" | jq -r '.userinfo_endpoint')

K8S: Configure Kubernetes API Server

We will setup minikube passing the configuration flags for the API Server directly from the command line.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# minikube setup
minikube start --driver docker --cpus 8 --memory max --profile minikube \
--embed-certs \
--extra-config=apiserver.authorization-mode=Node,RBAC \
--extra-config=apiserver.oidc-issuer-url=${OIDC_ISSUER_URL} \
--extra-config=apiserver.oidc-client-id=${OIDC_CLIENT_ID} \
--extra-config=apiserver.oidc-username-claim=name \
--extra-config=apiserver.oidc-username-prefix=- \
--extra-config=apiserver.oidc-groups-claim=groups \
--extra-config=apiserver.oidc-groups-prefix=

Command Line breakdown

— embed-certs : adds the certificates placed under $HOME/.minikube/certs/ — extra-config=apiserver.authorization-mode=Node,RBAC : this adds RBAC to the cluster while maintaining the local node access (to prevent lockout). — extra-config=apiserver.oidc-issuer-url=${OIDC_ISSUER_URL} : the issuer URL (which is the URL with the realm, or in case of another OpenID provider, the URL where you can then find the .well-known/openid-configuration — extra-config=apiserver.oidc-client-id=${OIDC_CLIENT_ID} : the client ID (in our case, the name of the client in the real) — extra-config=apiserver.oidc-username-claim=name : the username claim from the token. — extra-config=apiserver.oidc-username-prefix=- : intentionally configured with a single dash, which prevents any prefix to be appended to the name- — extra-config=apiserver.oidc-groups-claim=groups : the configuration for the group name. — extra-config=apiserver.oidc-groups-prefix= : this one is intentionally left blank to prevent the API Server to add any prefix (like - to the groups’ names)

Gotchas

The config is not yet ready or you have not configured an alternative to RBAC
The certificate is not trusted (because it is self-signed or it is not available in the trust store)

K8S: Configure RBAC

For now, we will create a ClusterRole and a ClusterRoleBinding to demonstrate how to create how to grant “ReadOnly” rights to Namespaces and Pods within the cluster:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


# rbac.yaml

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: k8s-ro
rules:
  - apiGroups: [""]
    resources: ["namespaces","pods"]
    verbs: ["get", "watch", "list"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: k8s-ro
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: k8s-ro
subjects:
- kind: Group
  name: "k8s"
  apiGroup: rbac.authorization.k8s.io

How does this work?

The ClusterRole defines which operations can be performed on the API Server and the ClusterRoleBinding matches the role with a specific subject: notice the subjects code block:

It references the name k8s of kind Group

By assigning this group (“k8s”) to our users they will get RO access to the clusters’ namespaces and pods.

Apply the config using the current local node credentials:

kubectl apply -f rbac.yaml Configure the Client (kubectl)

As explained in the introductory section, we will first obtain an ID token and then configure kubectl with it.

For this purpose, I will create a user called k8suser in Keycloak and assign it the group k8s . You can skip this part if you have other users and or groups, just adjust to your needs:

Preparing the command line to request the ID Token

For this purpuse I will use curl and jq to get a response from Keycloak.

I will then extract the tokens and use them to configure kubectl .

1
2
3


# Prepare some credentials
export K8S_USER='k8suser'
export K8S_USER_PASS='THE_USER_PASS'

1
2
3
4
5
6
7
8


export RESPONSE=$(curl -v -k -X POST \
-H "Content-Type: application/x-www-form-urlencoded" \
"${OIDC_TOKEN_ENDPOINT}" \
-d grant_type=password \
-d client_id=${OIDC_CLIENT_ID} \
-d username=${K8S_USER} \
-d password=${K8S_USER_PASS} \
-d scope="openid profile email name groups" | jq '.')

Extract the tokens:

1
2
3


export ID_TOKEN=$(echo $RESPONSE| jq -r '.id_token')
export REFRESH_TOKEN=$(echo $RESPONSE| jq -r '.refresh_token')
export ACCESS_TOKEN=$(echo $RESPONSE| jq -r '.access_token')

Now we can optionally check if we received all the information that we need by querying the user info endpoint with the access token.

Afterwards we do not need the access token anymore and we could discard it.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


curl -s -H "Content-Type: application/x-www-form-urlencoded" \
-H "Authorization: Bearer ${ACCESS_TOKEN}" \
"${OIDC_USERINFO_ENDPOINT}" | jq '.'

{
  "sub": "a42c4585-916d-4e9e-a068-ba1bbceb8887",
  "email_verified": true,
  "name": "k8suser",
  "groups": [
    "k8s"
  ],
  "preferred_username": "k8suser",
  "given_name": "k8suser",
  "family_name": "KC",
  "email": "k8suser@rieragalm.es"
}

Note: Check that name and groups appear and they are populated properly!

Gotchas

You forgot to make the claims default or you are not requesting those scopes, so there are not available in the ID Token

Kubectl: create a set of credentials

Now that we have the tokens, let’s create a new set of kubectl credentials!

1
2
3
4
5
6


kubectl config set-credentials ${K8S_USER} \
   --auth-provider=oidc \
   --auth-provider-arg=idp-issuer-url=${OIDC_ISSUER_URL} \
   --auth-provider-arg=client-id=${OIDC_CLIENT_ID} \
   --auth-provider-arg=refresh-token=${REFRESH_TOKEN} \
   --auth-provider-arg=id-token=${ID_TOKEN}

Now we have to create a new configuration context to use this newly created user in an existing cluster:

1
2
3
4


kubectl config set-context ${K8S_USER} \
--cluster=minikube \
--user=${K8S_USER} \
--namespace=default

And finally we need to use it:

1

kubectl config use-context ${K8S_USER}

Kubectl: check the access

We will now check our identity and we will try to perform a get operation to get the pods on the cluster:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


kubectl auth whoami
ATTRIBUTE   VALUE
Username    k8suser
Groups      [k8s system:authenticated]


kubectl auth can-i get pods
yes

kubectl auth can-i get deployments.apps
no

1
2
3
4
5
6
7
8
9


ubectl get pods -A
NAMESPACE     NAME                                      READY   STATUS    RESTARTS        AGE
kube-system   coredns-5dd5756b68-7xkmd                  1/1     Running   2 (2d18h ago)   2d23h
kube-system   etcd-minikube-docker                      1/1     Running   3 (2d18h ago)   2d23h
kube-system   kube-apiserver-minikube-docker            1/1     Running   1 (2d18h ago)   2d22h
kube-system   kube-controller-manager-minikube-docker   1/1     Running   3 (2d18h ago)   2d23h
kube-system   kube-proxy-cbjf9                          1/1     Running   2 (2d18h ago)   2d23h
kube-system   kube-scheduler-minikube-docker            1/1     Running   3 (2d18h ago)   2d23h
kube-system   storage-provisioner                       1/1     Running   5 (2d18h ago)   2d23h

1
2


kubectl get deployments.apps
Error from server (Forbidden): deployments.apps is forbidden: User "k8suser" cannot list resource "deployments" in API group "apps" in the namespace "default"

Kubernetes Automated Cluster Scaling

Thu, 17 Apr 2025 00:00:00 +0000

Kubernetes Automated Cluster Scaling

algorithm

AutoScaling is one of the most powerful concepts in Kubernetes.

This involves two main mechanisms:

Horizontal Pod Autoscaling (HPA)
Vertical Pod Autoscaling (VPA).

Automated cluster scaling refers to the process of dynamically adjusting the number of running pods (HPA) or their resource allocations (VPA) based on real-time metrics. This ensures that your applications can efficiently handle varying loads without manual intervention.

With HPA, you can scale smarter, and with VPA, scale wiser. HPA handles traffic spikes like a champ. VPA makes sure your pods get the resources they deserve.

Horizontal Pod Autoscaling (HPA)

HPA automatically adjusts the number of pods in a deployment or replica set based on observed CPU utilization or other select metrics. For instance, if your application experiences a sudden increase in traffic, HPA will scale out by adding more pods to handle the load. Conversely, it will scale in by reducing the number of pods when the load decreases.

Manual Scaling (Quick Recap). You manually scale pods using:

1

kubectl scale deployment  --replicas=5

Limitations: Manual, not reactive to load → not ideal for production.
Metrics Used: CPU/Memory utilization (via metrics-server).
Common Use Case: Web app getting heavy traffic → HPA increases pods → load balanced across more pods → better performance.
Key Fields in HPA YAML:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: my-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: 2
  maxReplicas: 10
  targetCPUUtilizationPercentage: 50

Deploying the Metrics Server

Why: HPA needs CPU/memory stats → metrics-server collects and exposes these.

Install:

1
2


kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
kubectl get deployment metrics-server -n kube-system

Create Pod + Service Example YAML:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36


apiVersion: apps/v1
kind: Deployment
metadata:
  name: loadapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: loadapp
  template:
    metadata:
      labels:
        app: loadapp
    spec:
      containers:
      - name: app
        image: k8s.gcr.io/hpa-example
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: 200m
          limits:
            cpu: 500m
---
apiVersion: v1
kind: Service
metadata:
  name: loadapp-svc
spec:
  selector:
    app: loadapp
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

Deploy the HPA

1
2


kubectl autoscale deployment loadapp --cpu-percent=50 --min=1 --max=10
kubectl get hpa

Simulate Load

To make CPU usage spike and trigger autoscaling:

1

kubectl run -i --tty load-generator --image=busybox /bin/sh

Inside the pod:

1

while true; do wget -q -O- http://loadapp-svc.default.svc.cluster.local; done

This sends continuous traffic to the service, increasing CPU usage.

Observe Scaling

1

watch kubectl get hpa

You’ll see replicas increasing as CPU crosses threshold (e.g., >50%).

1

kubectl get pods

Eventually:

More pods created
CPU usage spread across them
Load goes down

Vertical Pod Autoscaling (VPA)

VPA adjusts the resource requests and limits for your containers based on their usage. This means it can increase the CPU and memory allocated to a pod if it is consistently using more resources than initially requested, or it can reduce these allocations if the pod is over-provisioned.

What it does: Changes the resources (CPU, memory) allocated to each pod.
Use Case: Workloads where replica count doesn’t need to change, but need more resources.
Limitations: VPA restarts pods to apply changes.
Example YAML:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: my-vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: loadapp
  updatePolicy:
    updateMode: "Auto"

Install the VPA components from their official GitHub if not available in your cluster.

Tolerance

doc

version: Kubernetes v1.33

Tolerances appear under the spec.behavior.scaleDown and spec.behavior.scaleUp fields and can thus be different for scale up and scale down. A typical usage would be to specify a small tolerance on scale up (to react quickly to spikes), but higher on scale down (to avoid adding and removing replicas too quickly in response to small metric fluctuations).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-app
spec:
  ...
  behavior:
    scaleDown:
      tolerance: 0.05
    scaleUp:
      tolerance: 0

Kubernetes Controllers

Thu, 17 Apr 2025 00:00:00 +0000

What is a Replication Controller?

A Replication Controller (RC) ensures that a specified number of pod replicas are running at all times. It continuously monitors the cluster and if a pod fails, it will replace it to maintain the desired state.

Features

Ensures availability of pods by replacing failed ones.
Basic scaling of pods by changing the replica count.
Legacy Component: Being replaced by ReplicaSets in modern Kubernetes due to limited functionality.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


apiVersion: v1
kind: ReplicationController
metadata:
  name: my-rc
spec:
  replicas: 3
  selector:
    app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: nginx
        image: nginx:latest

1

kubectl get rc

What is a ReplicaSet?

A ReplicaSet (RS) is the newer, more advanced version of Replication Controllers. It provides additional functionality and is often managed by Deployments. Features

Supports set-based label selectors, allowing complex filtering of pods.
Works seamlessly with Deployments for rolling updates and rollbacks.
Dynamic scaling support by modifying the number of replicas.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: my-rs
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: nginx
        image: nginx:latest

1

kubectl get rs

What is a Deployment?

A Deployment is the most common way to manage ReplicaSets. It provides features like rolling updates, rollbacks, and scaling. Features

Manages ReplicaSets to maintain desired state of applications.
Supports Rolling Updates & Rollbacks to ensure zero downtime during updates.
Automatically scales pods based on defined replicas.
Provides self-healing capabilities to replace unhealthy pods.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: nginx
        image: nginx:latest

Check Deployment Status:

kubectl rollout status deployment/my-deployment

View Deployment History:

kubectl rollout history deployment/my-deployment

Rollback to Previous Version:

kubectl rollout undo deployment/my-deployment

Scale the Deployment:

kubectl scale deployment my-deployment –replicas=5

Summary

Replication Controller: Ensures availability of a specified number of pods.
ReplicaSet: Improved version of RC with better label selection and scaling support.
Deployment: Manages ReplicaSets, supports rolling updates, rollbacks, and scaling.

Kubernetes Init Containers

Thu, 17 Apr 2025 00:00:00 +0000

Kubernetes Init Containers

Kubernetes init containers are specialized containers that run to completion before any of your application’s primary containers start. Unlike regular containers, they are not part of your ongoing workload but instead perform initialization tasks such as setting up prerequisites, configuring environments, or fetching secrets. This ensures that the main containers only start when the system is fully prepared.

Isolation of Setup Tasks: They allow you to separate initialization logic from the main application, keeping your application images lean and secure.
Different Resource Allocation: Init containers may require different CPU/memory limits. The effective pod resource requests are determined by the highest values among the init containers and the app containers.

Using init containers offers several strategic advantages:

Enhanced Security: They can run privileged tasks (like fetching sensitive secrets from Vault or AWS Secrets Manager) without bloating the main application container image.
Environment Preparation: Init containers perform setup tasks like configuring databases, creating directories, or cloning repositories, ensuring that all dependencies are ready.
Simpler Application Images: By offloading initialization to separate containers, you can keep your main container images small, reducing the attack surface.
Better Resource Management: They allow you to allocate precise resources for initialization tasks that may have a short lifespan compared to the main application.

Example: Downloading Configuration from S3

When deploying a Django application on Kubernetes, it’s essential to run database migrations before starting the main application. If migrations are not applied, the app might crash due to missing tables or outdated schemas.

Using an init container, we can apply migrations before starting the Django web server. This ensures that:

The database schema is updated before the app runs.
The main container only starts after the migration process is successful.
There are no race conditions where multiple pods try to run migrations simultaneously.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41


apiVersion: apps/v1
kind: Deployment
metadata:
  name: django-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: django
  template:
    metadata:
      labels:
        app: django
    spec:
      initContainers:
      - name: run-migrations
        image: my-django-app:latest  # Use the same image as your main app
        command: ["python", "manage.py", "migrate"]
        env:
          - name: DATABASE_URL
            value: "postgres://user:password@postgres-service:5432/mydb"
        volumeMounts:
          - name: django-config
            mountPath: /app/config  # If using external config

      containers:
      - name: django
        image: my-django-app:latest
        command: ["gunicorn", "myproject.wsgi:application", "--bind", "0.0.0.0:8000"]
        ports:
          - containerPort: 8000
        env:
          - name: DATABASE_URL
            value: "postgres://user:password@postgres-service:5432/mydb"
        volumeMounts:
          - name: django-config
            mountPath: /app/config

      volumes:
      - name: django-config
        emptyDir: {}

How It Works

Init Container (run-migrations)
- Uses the same Django application image as the main container.
- Runs python manage.py migrate to apply database migrations.
- Ensures that the database is ready before the main application starts.
Main Container (django)
- Runs Gunicorn as the Django web server.
- Starts only after the init container completes successfully.
Why Use an Init Container for Migrations?
- Prevents multiple app instances from running migrations simultaneously.
- Guarantees that migrations are applied before the app starts.
- Ensures a stable startup process for Django in Kubernetes.

Advanced Insights and Unknown Facts

Resource Calculation for Init Containers

Effective Resource Requests: The highest resource request (CPU/memory) specified among all init containers becomes the effective request for the pod. This might impact scheduling, so plan resource allocation carefully.

Native Sidecar Functionality

Alpha Feature: Kubernetes v1.28 introduced native sidecar support via init containers by setting restartPolicy: Always. This allows an init container to function as a persistent sidecar that runs alongside your main application.
Use Case: Ideal for logging agents or monitoring tools that need to run continuously without blocking pod termination.

Best Practices

Keep It Focused: Design init containers to perform a single, well-defined task.
Idempotency: Since init containers may be retried, ensure your initialization logic is idempotent.
Monitor Logs: Even after init containers finish, their logs are available for debugging purposes.
Security: Avoid embedding sensitive credentials in the main container image; fetch them securely in an init container instead.

Helm: The package manager for Kubernetes

Sun, 13 Apr 2025 00:00:00 +0000

Helm

helm fetch

You can use helm fetch to Download a chart to your local directory, so You can change the values in values.yaml file and then install it.

for example:

`1`	`helm fetch stable/superset --untar`

Kubernetes Gateway Api

Sun, 13 Apr 2025 00:00:00 +0000

The Kubernetes Gateway API

The Kubernetes Gateway API is a modern, extensible standard for managing ingress and routing traffic in Kubernetes environments. It builds upon the limitations of the legacy Ingress API to provide a vendor-agnostic, declarative framework for configuring L4 and L7 network traffic. The Gateway API is designed to unify and simplify traffic management while supporting advanced use cases such as multi-tenancy, path-based routing, and traffic splitting.

Vendor-Agnostic Abstraction:

The Gateway API provides a universal standard for ingress and traffic routing, eliminating the dependency on vendor-specific annotations.

It allows seamless migration across cloud providers and ingress controllers without major changes to the configuration.

Separation Of Concerns:

Enables clear boundaries between application teams and infrastructure teams by splitting responsibilities -

Infrastructure Teams: Manage the Gateway resources (e.g., ingress controllers, load balancers).

Application Teams: Define Routes (e.g., HTTPRoute, TCPRoute) without worrying about the underlying network infrastructure.

Layer 7 Load Balancing:

Offers advanced routing capabilities, including path-based routing, header-based routing, and traffic splitting for canary deployments or blue-green deployments.

Supports precise L7 policies like rate limiting, request rewrites, and cross-origin resource sharing (CORS).

Extensibility:

Designed with extensibility in mind, it allows custom implementations via GatewayClass, enabling features specific to vendors like AWS, GCP, or Istio while adhering to the Gateway API standard.

Standardized CRDs reduce reliance on annotations.

How The Gateway API Solves The “Annotation Chaos”

The legacy Ingress API relied on annotations to configure features like SSL termination, load balancing algorithms, or backend timeouts. Each ingress controller implemented annotations differently, causing inconsistencies across environments.

The Gateway API eliminates this problem by introducing Custom Resource Definitions (CRDs) for defining

Gateways: Representing the physical or logical entry point (e.g., load balancers, proxies).

Routes: Defining application-specific routing rules (e.g., HTTPRoute, TCPRoute).

Lack Of Flexibility:

The Ingress API was designed as a one-size-fits-all solution but fails to cater to advanced use cases. It offers limited configurability for Layer 7 (L7) routing beyond basic HTTP/HTTPS functionality.

Heavy Reliance On Annotations:

To extend Ingress capabilities, vendors introduced annotations, often in incompatible ways. For instance, annotations for AWS ALB differ significantly from those for NGINX or Traefik. This reliance creates vendor lock-in, making migrations between platforms challenging.

Multi-Layer Traffic Routing:

Applications often require both external ingress (e.g., handling traffic from the internet) and internal service-to-service communication. Routing traffic efficiently across these layers requires features like -

Path-based and header-based routing.

Weighted traffic splitting for canary deployments or A/B testing.

Mutual TLS (mTLS):

Securing communication between microservices is critical, especially for sensitive workloads. Traditional networking setups often struggle to enforce mTLS consistently across clusters.

Layer 4 (L4) & Layer 7 (L7) Traffic Management:

Applications frequently require both low-level TCP/UDP routing (L4) and high-level HTTP/S routing (L7). Traditional setups struggle to provide unified solutions that handle both effectively.

Verify Gateway version: kubectl get crd gateways.gateway.networking.k8s.io -o yaml

CRDs update: kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v0.7.0/gateway.networking.k8s.io.crds.yaml

Upgrade Talos Linux and Kubernetes

Sun, 13 Apr 2025 00:00:00 +0000

Upgrade Kubernetes Version

1

$ talosctl --nodes 192.168.1.71 etcd snapshot etcd.backup

Upgrading Kubernetes is non-disruptive to the cluster workloads.

You can do this live, assuming you don’t have single-replica workloads that are node-specific.

Today I will be upgrading to Kubernetes version v1.31.5. I’m currently on v1.30.0 but I want to make sure I’m running the same version that is being tested on the CKA exam that I’m studying for which is currently 1.31.

Talos recommends using the talosctl upgrade-k8s command which automatically upgrades the entire cluster and has built in safety checks. They explain how to do it manually, but I chose Talos Linux partly based on the ease of ongoing maintenance and upgrades so I will be using the easy button here!

Check current version: kubectl get node
Upgrade: talosctl -n 10.0.50.11 upgrade-k8s --to 1.31.5
- You only need to specify a single control plane node, but this will upgrade the whole cluster
- You need to choose a real Kubernetes version - https://kubernetes.io/releases/
- This will take a while, so try to be patient.
Verify version: kubectl get node

I like to update my local talosconfig repo which was used to deploy the original Talos cluster and also includes secrets used to recover in case of any problems. This is a good time to update the Kubernetes version in controlplane.yaml and worker.yaml for any new nodes you deploy.

Upgrade Talos OS

The Talos team recommends using the same version of talosctl that your nodes are running. You will then upgrade talosctl after the node upgrades are complete.

Be sure to upgrade one node at a time and check that it’s healthy before moving on. You can blast through them using a for loop, or do them by hand. Just don’t do them all at the same time :)

Check versions:
- Client: talosctl version --client
- Server: kubectl get node -o wide (OS-IMAGE column)
Get a new Talos OS image from the factory: https://factory.talos.dev
- Make sure to add any existing extensions you’re using such is iscsi-tools
- Copy the image string under the “Upgrading Talos Image” header. In my case this looks like factory.talos.dev/installer/88d1f7a5c4f1d3aba7df787c448c1d3d008ed29cfb34af53fa0df4336a56040b:v1.9.2
Upgrade one node: talosctl upgrade -n 10.0.50.11 --image factory.talos.dev/installer/88d1f7a5c4f1d3aba7df787c448c1d3d008ed29cfb34af53fa0df4336a56040b:v1.9.2 --preserve
- -n: Specify the node to upgrade
- --image: Specify the factory image to use
- --preserve: Don’t wipe extraMounts if applicable. I default to using this unless I have a specific reason to wipe additional mounts.
Repeat the upgrade command for each node, one at a time, until all nodes have been upgraded.

1

$ talosctl --talosconfig ~/.talos/talosconfig upgrade -n node3 --image factory.talos.dev/installer/88d1f7a5c4f1d3aba7df787c448c1d3d008ed29cfb34af53fa0df4336a56040b:v1.9.2 --preserve

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


➜  talosctl -n 192.168.1.71 version

Client:
        Tag:         v1.7.6
        SHA:         ae67123a
        Built:
        Go version:  go1.22.5
        OS/Arch:     darwin/amd64
Server:
        NODE:        192.168.1.71
        Tag:         v1.7.6
        SHA:         ae67123a
        Built:
        Go version:  go1.22.5
        OS/Arch:     linux/arm64
        Enabled:     RBAC


❯  talosctl upgrade --nodes 192.168.1.71 --image ghcr.io/siderolabs/installer:v1.9.5
◲ watching nodes: [192.168.1.71]
    * 192.168.1.71: task: removeAllPods action: START

In my homelab, I am comfortable blasting through upgrades with a for loop, so my upgrade command looks like this:

1

for node in 11 12 13 21 22 23 31 32 33; do talosctl upgrade -n 10.0.50.$node --image factory.talos.dev/installer/88d1f7a5c4f1d3aba7df787c448c1d3d008ed29cfb34af53fa0df4336a56040b:v1.9.2 --preserve; done

Once Nodes have been upgraded, upgrade the talosctl client so the version matches the Talos node version.

rm /usr/local/bin/talosctl
curl -sL https://talos.dev/install | sh (this gets the latest version)
- You can also download a specific release from https://github.com/siderolabs/talos/releases, e.g. curl -LJO https://github.com/siderolabs/talos/releases/download/v1.8.3/talosctl-linux-amd64
Verify: talosctl version --client

I like to update my local talosconfig repo which was used to deploy the original Talos cluster and also includes secrets used to recover in case of any problems. This is a good time to update the factory image in controlplane.yaml and worker.yaml for any new nodes you deploy.

1
2
3
4
5
6
7


$ talosctl -n node1 health
discovered nodes: ["192.168.1.71" "192.168.1.59" "192.168.1.73"]
waiting for etcd to be healthy: ...
waiting for etcd to be healthy: OK
waiting for etcd members to be consistent across nodes: ...
waiting for etcd members to be consistent across nodes: OK
...

1

➜  talosctl --nodes 192.168.1.71 dmesg -f

Kubernetes: CRD, Custom Resource Definition

Sat, 05 Apr 2025 00:00:00 +0000

Create your own CRD

CRD (Custom Resource Definition)

This defines how we want our custom Kubernetes objects to look and behave.

The name of the CRD must follow this format: .

Example: albertocrds.crds.albertogalvez.com

To list existing CRDs:

`1`	`kubectl get crds`

In the custom resource manifest, you must specify:

1
2
3

apiVersion: crds.albertogalvez.com/v1
kind: Albertocrds
...

Controller

Now we create our controller, which is simply an application running continuously in Kubernetes, listening for changes to our custom resources.

Kubectl: Kubernetes Swiss Knife

Sat, 29 Mar 2025 00:00:00 +0000

Kubectl

kubectl taint

Taints are used to prevent pods from being scheduled on certain nodes unless those pods have the appropriate tolerations.

1

kubectl taint nodes node1 node2 node3 node-role.kubernetes.io/control-plane:NoSchedule-

node-role.kubernetes.io/control-plane:NoSchedule is a taint typically applied to control plane nodes to prevent regular pods from being scheduled on them.
The - at the end removes the taint from the nodes.

If you want to add the taint instead of removing it, just remove the - at the end:

1

kubectl taint nodes node1 node2 node3 node-role.kubernetes.io/control-plane:NoSchedule

kubectl wait

1

kubectl wait -n istio-ingress --for=condition=programmed gateways.gateway.networking.k8s.io gateway

This kubectl command waits for a specific condition (in this case, for the Gateway resource to be programmed) within a namespace called istio-ingress. Here’s a breakdown of each part:

kubectl wait: This command pauses execution until a resource or set of resources meets a specified condition.
-n istio-ingress: Specifies the namespace where the resource is located. In this case, the istio-ingress namespace.
–for=condition=programmed: Defines the condition that must be met. Here, it means the Gateway resource must reach the programmed state—indicating it has been properly configured and is ready for use.
gateways.gateway.networking.k8s.io: Specifies the type of resource being monitored. In this case, a Gateway resource from the gateway.networking.k8s.io API group.
gateway: The name of the specific Gateway resource you’re monitoring.

What does this mean in practice?

The command waits until the Gateway resource in the istio-ingress namespace is fully configured (i.e., programmed) and ready to handle network traffic. This is especially useful when you need to ensure that a network configuration or proxy (like an Istio Gateway) is ready before proceeding with a deployment or any other dependent operations.

If the Gateway does not reach the specified condition within the default timeout period (30 seconds by default, which can be adjusted using –timeout), the command will fail.

Plugins

Basic Structure of a kubectl Plugin

kubectl plugins are simply executables that start with the kubectl- prefix. When you execute a command like:

kubectl my-plugin

kubectl searches for an executable called kubectl-my-plugin in the directories listed in the PATH environment variable and executes it.

k9scli: Kubernetes cli to manage your clusters

Wed, 12 Mar 2025 00:00:00 +0000

K9scli

Shortcuts

Ctrl-d: delete.
Ctrl-k: kill (no confirmation).
Ctrl-w: toggle wide columns. (Equivalent to kubectl … -o wide)
Ctrl-z: toggle error state
Ctrl-e: hide header.
Ctrl-s: save output (e.g. the YAML) to disk.
Ctrl-l: rollback.

Sort by Column

If you want to sort any view (Pod/Services) based on some exact column - you can just press Shift + Column Initial

e.g. If you want to sort items by column Age - Just press Shift + A

Shift-c: sorts by CPU.
Shift-m: sorts by MEMORY.
Shift-s: sorts by STATUS.
Shift-n: sorts by name;
Shift-o: sorts by node;
Shift-i: sorts by IP address;
Shift-a: sorts by container age;
Shift-t: sorts by number of restarts;
Shift-r: sorts by pod readiness;

List all available resources:

:aliases or Ctrl-a: list all available aliases and resources. :crd: list all CRDs.

Filter

/ : regex filter.
/! : inverse regex filter.
/-l : filter by labels.
/-f : fuzzy match.

Choose context

:ctx: list ctx, then select from the list. :ctx : switch to the specified context.

Show Decrypted Secrets

Type :secrets to list the secrets, then

x to decrypt the secret.
Esc to leave the decrypted display.

x: decode a Secret. f: full screen. Tip: enter full screen mode before copying, to avoid in copied text.

Helm

:helm : show helm releases.
:helm NAMESPACE : show releases in a specific namespace.

XRay View

:xray RESOURCE , e.g. :xray deploy.

Pulse View

:pulse : displays general information about the Kubernetes cluster.

Show Disk Files

:dir /path

E.g. :dir /tmp will show your /tmp folder on local disk. One common use case: Ctrl-s to save a yaml, then find it in :dir /tmp/k9s-screens-root, find the file, press e to edit and a to apply.

Benchmark

k9s includes a basic HTTP load generator.

To enable it, you have to configure port forwarding in the pod. Select the pod and press SHIFT + f, go to the port-forward menu (using the pf alias).

After selecting the port and hitting CTRL + b, the benchmark would start. Its results are saved in /tmp for subsequent analysis.

To change the configuration of the benchmark, create the $HOME/.k9s/bench-.yml file (unique for each cluster).

Check Resources with the Same Name in Different API Groups

e.g. Cluster may be found in different api groups, like cluster.x-k8s.io or clusterregistry.k8s.io or baremetal.cluster.gke.io.

1
2
3
4
5
6
7
8


apiVersion: cluster.x-k8s.io/v1alpha3
kind: Cluster

apiVersion: clusterregistry.k8s.io/v1alpha1
kind: Cluster

apiVersion: baremetal.cluster.gke.io/v1
kind: Cluster

Use apiVersion/kind (i.e. Group/Version/kind) instead of just kind to check the API of a specific group.

:cluster.x-k8s.io/v1alpha3/clusters
:clusterregistry.k8s.io/v1alpha1/clusters
:baremetal.cluster.gke.io/v1/clusters

Change log setting

Change ~/.config/k9s/config.yml:

logger:
  tail: 500
  buffer: 5000
  sinceSeconds: -1

CPU/MEM Metrics

A small change here based on Benjamin excellent PR! We’ve added 2 new columns for pod/container views to indicate percentages of resources request/limits if set on the containers. The columns have been renamed to represent the resources requests/limits as follows:

Name	Description	Sort Keys
%CPU/R	Percentage of requested cpu	shift-x
%MEM/R	Percentage of requested memory	shift-z
%CPU/L	Percentage of limited cpu	ctrl-x
%MEM/L	Percentage of limited memory	ctrl-z

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


View: Pods()[number of pods listed]

NAME      pod name
READY     number of pods in ready state / number of pods to be in ready state
RESTARTS  number of times the pod has been restarted so far
STATUS    state of the pod life cycle, such as Running | ... | Completed
CPU       current CPU usage, unit is milli-vCPU
MEM       current main memory usage, unit is MiB
%CPU/R    current CPU usage as a percentage of what has been requested by the pod
%MEM/R    current main memory usage as a percentage of what has been requested by the pod
%CPU/L    current CPU usage as a percentage of the pod's limit (it cannot go beyond its limit)
%MEM/L    current main memory usage as a percentage of the pod's limit (it cannot go beyond its limit)
IP        IP address of the pod
NODE      name of the node the pod is running on
AGE       age of the pod, units are indicated (s = seconds, m = minutes, h = hours, d = days)

What about CPU/A and MEM/A when you see the nodes?

CPU/A is about the CPU allocatable (unit is milli-vCPU)
MEM/A is the memory allocatable (unit is MiB)

if you’re asking yourself like me, what a milli-vCPU is 1/1000 of (Threads x Cores) x Physical CPU = Number vCPU

Views

Example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


views:
  v1/endpoints:
    columns:
      - AGE|RW
      - NAME
      - ENPOINTS|H
      - BLA:.subsets[*].ports[*].port
      - BOZO:.metadata.labels.app|W
      - BLEE:.metadata.creationTimestamp|T
      - ZORG:.status.containersStatuses[*].restart

keywords:
- W -> show only in wide mode.
- H -> hide.

Pod disruption budget (PDB)

Sun, 23 Feb 2025 00:00:00 +0000

Pod disruption budget

PDBs define the minimum number of replicas that must remain running during disruptions. This prevents critical workloads from being evicted but can hinder node scaling.

A PodDisruptionBudget (PDB) is a Kubernetes object that specifies the number of pods that can be unavailable in deployment, maintenance, or at any given time. This helps to ensure that your applications remain available even if some of their pods are terminated or evicted.

Let’s take an example where my application has three pods (instances); I always want to have at least two running pods all the time; I can apply a PDB object which will guarantee that I will always have at least two running pods!

1
2
3
4
5
6
7
8
9


apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-pdb
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: my-app

This configuration ensures one replica are always running but can block eviction when scaling down nodes, leaving some underutilized.

Kustomize: Kubernetes deployment

Sun, 16 Feb 2025 00:00:00 +0000

Kustomize

Kustomize is an official sub-project of Kubernetes and is maintained by the Kubernetes SIG-CLI (Special Interest Group – Command Line Interface) community. It has gained popularity and is widely used in the Kubernetes ecosystem.

Kustomize is an open-source configuration management tool for Kubernetes.

It allows you to define and manage Kubernetes objects such as deployments, Daemonsets, services, configMaps, etc for multiple environments in a declarative manner without modifying the original YAML files. To put it simply, you have a single source of truth for YAMLs, and you patch required configurations on top of the base YAMLs as per the environment requirements.

Kustomize has two key concepts, Base and Overlays. With Kustomize we can reuse the base files (common YAMLs) across all environments and overlay (patches) specifications for each of those environments.

Overlaying is the process of creating a customized version of the manifest file (base manifest + overlay manifest = customized manifest file).

Kustomize Features and Benefits

The following are the key features of Kustomize:

Acts as a configuration tool with declarative configuration same as Kubernetes YAMLs.
It can modify resources without altering the original files.
It can add common labels and annotations to all the resources.
It can Modify container images based on the environment it is being deployed in.
Kustomize also ships with secretGenerator and configMapGenerator that use environment files or key-value pairs to create secrets and configMaps.

Below are the benefits of Kustomize:

Simplified Configuration Management: Kustomize is easy to use and allows you to manage and customize your Kubernetes configurations more easily by enabling you to define your configuration in a structured and modular way.
Reusability: With Kustomize we can reuse one of the base files across all environments and overlay specifications for each of those environments. This can save you time and effort by allowing you to reuse common configurations rather than having to create them from scratch for each new deployment.
Version Control: Kustomize allows you to version control your Kubernetes configurations, making it easier to track changes and roll back to previous configurations if necessary.
Template Free: Kustomize is template free. It expresses the full power of Kubernetes API, with no need to parameterize every single line compared to Helm.
Kustomize can be run natively from the Kubernetes command line interface.
Kustomize has built-in transformers to modify resources and It can be extended via a plug-in mechanism.
Kustomize does not have any templating language so we can use the usual YAML to state our configurations rapidly.
Kustomize is provided as a standalone Golang package and cli tool so it’s easy to integrate with users’ tools and workflows.
We can use Kustomize without installing it if we have kubectl 1.14+ version. kubectl allows us to make declarative changes to our configurations without touching a template.

kustomization.yamlfile

The kustomization.yaml file is the main file used by the Kustomize tool.

When you execute Kustomize, it looks for the file named kustomization.yaml. This file contains a list of all of the Kubernetes resources (YAML files) that should be managed by Kustomize. It also contains all the customizations that we want to apply to generate the customized manifest.

Base and Overlays

The Base folder represents the config that going to be identical across all the environments. We put all the Kubernetes manifests in the Base. It has a default value that we can overwrite.

On the other side, the Overlays folder allows us to customize the behavior on a per-environment basis. We can create an Overlay for each one of the environments. We specify all the properties and parameters that we want to overwrite & change.

Basically, Kustomize uses patch directive to introduce environment-specific changes on existing Base standard k8s config files without disturbing them. Kustomize will check the base deployment file and compare it and patch the changes accordingly. That’s the beauty of Kustomize.

Transformers

As the name indicates, transformers are something that transforms one config into another. Using Transformers, we can transform our base Kubernetes YAML configs. Kustomize has several built-in transformers. Let’s see some common transformers:

commonLabel – It adds a label to all Kubernetes resources
namePrefix – It adds a common prefix to all resource names
nameSuffix – It adds a common suffix to all resource names
Namespace – It adds a common namespace to all resources
commonAnnotations – It adds an annotation to all resources

Let’s see an example. In the below image, we have used commonLabels in kustomization.yaml where label env: dev gets added to the customized deployment.yaml.

Image Transformer

It allows us to modify an image that a specific deployment is going to use.

In the following example, the image transformer checks the nginx image name as mentioned deployment.yaml and changes it to the new name which is ubuntu in the kustomization.yaml file. We can change the tags as well.

Patches (Overlays)

Patches or overlays provide another method to modify Kubernetes configs. It provides more specific sections to change in the configuration. There are 3 parameters we need to provide:

Operation Type: add or remove or replace
Target: Resource name which we want to modify
Value: Value name that will either be added or replaced. For the remove operation type, there would not be any value.

There are two ways to define the patch:

JSON 6902 and
Stragetic Merge Patching.

JSON 6902 Patching

In this way, there are two details that we have to provide, the target and the patch details i.e. operation, path, and the new value.

1
2
3
4
5
6
7
8


patches:
  - target:
      kind: Deployment
      name: web-deployment
    patch: |-
      - op: replace
        path: /spec/replicas
        value: 5

Stragetic Merge Patching

In this way, all the patch details are similar to a standard k8s config. It would be the original manifest file, we just add the fields that need to be modified.

1
2
3
4
5
6
7
8


patches:
  - patch: |-
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: web-deployment
      spec:
        replicas: 5

Patch From File

For both types of patching, instead of inline configs, we can use the separate file method. Specify all the patch details in a YAML file and refer it to the kustomization.yaml file under the patches directive.

For example, in kustomization.yaml you need to mention the patch file as follows. You need to specify the relative path of the YAML file.

1
2


patches:
- path: replicas.yaml

And we can put the changes in replicas.yaml as given below.

1
2
3
4
5
6


apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-deployment
spec:
  replicas: 5

Review & Apply Patches

Let’s review the patches. We can use the below command to review the patches and check whether everything is correct or not.

1

kustomize build overlays/dev

Deploy

We can deploy the customized manifest using the following command.

1

kustomize build overlays/dev | kubectl apply -f -

You can also use the following kubectl command.

1

kubectl apply -k overlays/dev

Kustomize Configmap and Secret Generators

Kustomize has the functionality to generate Configmaps and Secrets.

In Kustomization YAML there are two supported fields

configMapGenerator and
secretGenerator

Let’s understand what problem it solves.

When you update a configmap attached to a pod as a volume, the configmap data gets propagated to the pod automatically. However, the pod does not get the latest data in the configmap in the following scenarios.

If the pod gets environment variables from the configmap.
If the configmap is mounted as a volume using a subpath.

In the above cases, the pod will continue using the old configmap data until we restart the pod. Because the pod is unaware of what got changed in configMap.

Essentially, the data from the ConfigMaps (such as properties, environment variables, etc.) is used by applications during their startup. So even if the updated configmap data is projected to the pod, if the application running inside the pod doesn’t have any hot reload mechanism, you will have to restart the pod for the changes to take place.

What options do we have to solve this issue?

You can use Reloader Controller.
Using Kustomize ConfigMap Generator

How the Kustoimize Configmap/Secret generator work.

Kustomize generator creates a configMap and Secret with a unique name(hash) at the end. For example, if the name of the configmap is app-configmap, the generated one would have the name app-configmap-7b58b6ct6d. Here 7b58b6ct6d is the appended hash.
If you update the configmap/Secret, it will create a new configMap/Secret with the same name with a different hash(random sets of characters) at the end.
Kustomize will automatically update the Deployment with the new configmap name.
The moment Deployment is updated by Kustomize, a rollout will be triggered and the application runs on the pod and gets the updated configmap/secret data. In this way, we don’t need to redeploy or restart the deployment.

Following are the important points you should know about the Kustomize generators.

Since Kustomize creates a new configmap every time there is an update, you need to garbage-collect your old orphaned Configmaps. If you have resource quota limits set for namespace, orphaned Configmaps could be an issue. Or you should use the –prune flag with labels in the kubectl apply command. Also, GitOps tools like ArgoCD offer Orphaned resource monitoring mechanisms.
You can use the disableNameSuffixHash: true flag to disable creating new Configmaps on every update, but it does not trigger a pod rollout. You need to manually trigger a rollout for pods to get the latest configmap data. Or the application running inside the pod should have a hot-reload mechanism.

Here is the file structure of the repository. To understand the generators, we will use the generators overlay folder.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


├── base
│   ├── deployment.yaml
│   ├── kustomization.yaml
│   └── service.yaml
└── overlays
    ├── dev
    │   ├── deployment-dev.yaml
    │   ├── kustomization.yaml
    │   └── service-dev.yaml
    ├── generators
    │   ├── deployment.yaml
    │   ├── files
    │   │   └── index.html
    │   ├── kustomization.yaml
    │   └── service.yaml
    └── prod
        ├── deployment-prod.yaml
        ├── kustomization.yaml
        └── service-prod.yaml

The configmap generation options should be added to the kustomization.yaml file under configMapGenerator field.

If you want to prune the orphaned Configmaps, use the –prune flag with the configmap label as shown below. The –prune flag instructs Kustomize to remove any resources from the final output that is no longer referenced or required.

1

kustomize build overlays/generators | kubectl apply --prune -l app=web-service  -f -

Disabling Hashed ConfigMap

If you don’t want to create hashed Configmaps using the ConfigMap generator, you can disable it by setting the disableNameSuffixHash flag to true under generatorOptions. It will disable the hash for all the Configmaps mentioned in the kustomization.yaml file.

1
2
3
4


generatorOptions:
  labels:
    app: web-service
  disableNameSuffixHash: true

Note: If you disable Configmap hash, you need to manually restart the pods for the configmap data to be consumed by the application.

Generate Secrets Using Kustomize

You can generate secrets the same way you generate Configmaps.

For generating secrets, you need to use the secretGenerator field.

Here is an example of generating a secret object from a file.

1
2
3
4


secretGenerator:
- name: nginx-secret
  files:
  - files/secret.txt

If you want to generate secrets from literals, use the following format.

1
2
3
4


secretGenerator:
- name: nginx-api-password
  literals:
  - password="myS3cret"

You can mount the secret as a volume or propagate it as an environment variable as per your requirements.

Troubleshooting

While developing or before pushing to git, run kubectl kustomize cfg fmt file_name command to format the file and set the indentation right.

Installing talos on a Turing Piv2 board with cm4 modules

Sun, 05 Jan 2025 00:00:00 +0000

Installing talos on a Turing Piv2 board with cm4 modules.

Flash image to all nodes

download metal-arm64.raw.xz from https://github.com/siderolabs/talos/releases
after putting cm4 in usb mode (use rpi cm4 emmc usb programming stick or use tpiv2, see: https://help.turingpi.com/hc/en-us/articles/8687165986205-Install-OS)

flash talos, this works for sd card and emmc storage. Check the correct device name.

time xz -d < metal-arm64.raw.xz | sudo dd of=/dev/sda bs=1M status=progress conv=fsync
893665280 bytes (894 MB, 852 MiB) copied, 3 s, 298 MB/s1306525696 bytes (1,3 GB, 1,2 GiB) copied, 3,82763 s, 341 MB/s

0+152937 records in
0+152937 records out
1306525696 bytes (1,3 GB, 1,2 GiB) copied, 99,591 s, 13,1 MB/s

real	1m39,600s
user	0m3,607s
sys	0m0,226s

make sure tpiv4 node is no longer in device mode, it should be host mode for normal operation.
optionally log in to bmc and connect to serial console of the node with minicom.

Minicom example for node2

1

microcom -s 115200 /dev/ttyS1

Hardwired bmc serial port connections to nodes

Node	bmc device
Node 1	/dev/ttyS2
Node 2	/dev/ttyS1
Node 3	/dev/ttyS4
Node 4	/dev/ttyS5

Boot nodes

boot all the nodes by powering down and up using the bmc
Check talos’ API port

1

nc -zv 192.168.1.71 50000

Create Cluster

Generate cluster files

Generate secrets:

1

talosctl gen secrets -o secrets.yaml

Generate control and worker config:

Caveats (according talos documentation):

Since VIP functionality relies on etcd for elections, the shared IP will not come alive until after you have bootstrapped Kubernetes. Don’t use the VIP as the endpoint in the talosconfig, as the VIP is bound to etcd and kube-apiserver health, and you will not be able to recover from a failure of either of those components using Talos API.

1
2


talosctl gen config --with-secrets secrets.yaml "clustername" https://CONTROL_PLANE_IP:6443
talosctl gen config --with-secrets secrets.yaml "comanche" https://192.168.1.71:6443

Edit controlplane.yaml:

set controlPlane.scheduler.disabled: false, I want control plane nodes to schedule work.
set interface with the value output from talosctl -n node3 get links
add the VIP ipadress to network stanza:

I have:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


      network:
        hostname: node3
        interfaces:
        - dhcp: true
          interface: enxd83addbb1c3a
          vip:
            ip: 192.168.1.59
        nameservers:
        - 8.8.8.8
        - 8.8.4.4

set install.disk: to /dev/mmcblk0
optionally set install.wipe: true

For each node (I have 3, an uneven number of control nodes is recommended in k8s):

change hostname: network.hostname: nodeX (set X to cm4 number)

apply config to cm4:

1
2
3
4
5
6
7


talosctl apply-config --insecure -n  --file controlplane.yaml
talosctl apply-config --insecure -n 192.168.1.71 --file controlplane-node1.yaml
talosctl apply-config --insecure -n 192.168.1.72 --file controlplane-node2.yaml
talosctl apply-config --insecure -n 192.168.1.73 --file controlplane-node3.yaml

[  118.061987] [talos] etcd is waiting to join the cluster, if this node is the first node in the cluster, please run `talosctl bootstrap` against one of the following IPs:
[  118.080674] [talos] [192.168.1.71]

Bootstrap cluster by one of the control nodes.

1
2
3


   talosctl  -n  -e  --talosconfig ./talosconfig bootstrap
talosctl --talosconfig ~/.talos/talosconfig -n 192.168.1.71 -e 192.168.1.71 bootstrap
[  259.296059] [talos] bootstrap request received

Grab ☕

Generate kubeconfig.

1
2
3
4


talosctl kubeconfig -f -n 

unset KUBECONFIG
talosctl kubeconfig --talosconfig talosconfig -n 192.168.1.71 -e 192.168.1.71

Watch cluster

watch -n 1.5 kubectl --kubeconfig=./kubeconfig --request-timeout=1s get pods,deployment,services,nodes -A -o wide

talosctl config nodes 192.168.1.71,192.168.1.72,192.168.1.73 –talosconfig ~/.talos/talosconfig talosctl config endpoint 192.168.1.71,192.168.1.72,192.168.1.73 –talosconfig ~/.talos/talosconfig

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


❯ talosctl -n node1 health

discovered nodes: ["192.168.1.59" "192.168.1.72" "192.168.1.73"]
waiting for etcd to be healthy: ...
waiting for etcd to be healthy: OK
waiting for etcd members to be consistent across nodes: ...
waiting for etcd members to be consistent across nodes: OK
waiting for etcd members to be control plane nodes: ...
waiting for etcd members to be control plane nodes: OK
waiting for apid to be ready: ...
waiting for apid to be ready: OK
waiting for all nodes memory sizes: ...
waiting for all nodes memory sizes: OK
waiting for all nodes disk sizes: ...
waiting for all nodes disk sizes: OK
waiting for kubelet to be healthy: ...
waiting for kubelet to be healthy: OK
waiting for all nodes to finish boot sequence: ...
waiting for all nodes to finish boot sequence: OK
waiting for all k8s nodes to report: ...
waiting for all k8s nodes to report: OK
waiting for all k8s nodes to report ready: ...
waiting for all k8s nodes to report ready: OK
waiting for all control plane static pods to be running: ...
waiting for all control plane static pods to be running: OK
waiting for all control plane components to be ready: ...
waiting for all control plane components to be ready: OK
waiting for kube-proxy to report ready: ...
waiting for kube-proxy to report ready: OK
waiting for coredns to report ready: ...
waiting for coredns to report ready: OK
waiting for all k8s nodes to report schedulable: ...
waiting for all k8s nodes to report schedulable: OK

1
2
3
4
5
6
7


❯ talosctl --talosconfig ~/.talos/talosconfig config info

Current context:     comanche
Nodes:               192.168.1.71,192.168.1.72,192.168.1.73
Endpoints:           192.168.1.71,192.168.1.72,192.168.1.73
Roles:               os:admin
Certificate expires: 1 year from now (2025-12-31)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


❯ talosctl -n node1 get admissioncontrolconfigs.kubernetes.talos.dev admission-control -o yaml

node: node1
metadata:
    namespace: controlplane
    type: AdmissionControlConfigs.kubernetes.talos.dev
    id: admission-control
    version: 1
    owner: k8s.ControlPlaneAdmissionControlController
    phase: running
    created: 2025-01-01T19:14:12Z
    updated: 1970-01-01T00:00:09Z
spec:
    config:
        - name: PodSecurity
          configuration:
            apiVersion: pod-security.admission.config.k8s.io/v1alpha1
            defaults:
                audit: restricted
                audit-version: latest
                enforce: baseline
                enforce-version: latest
                warn: restricted
                warn-version: latest
            exemptions:
                namespaces:
                    - kube-system
                runtimeClasses: []
                usernames: []
            kind: PodSecurityConfiguration

1

❯ talosctl -n node1 logs kubelet

1
2
3
4


❯ talosctl patch mc --talosconfig talosconfig --nodes 192.168.1.71 -e 192.168.1.71 --patch @patch.yaml
WARNING: 192.168.1.71: server version 1.7.6 is older than client version 1.9.1
patched MachineConfigs.config.talos.dev/v1alpha1 at the node 192.168.1.71
Applied configuration without a reboot

Client installation

MacOS:

1
2
3
4


curl -LO https://github.com/siderolabs/talos/releases/download/v1.7.6/talosctl-darwin-amd64
chmod +x talosctl-darwin-amd64
sudo mv talosctl-darwin-amd64 /usr/local/bin/talosctl
talosctl version

Operations

Show machine configuration:

1

talosctl -n node3 get machineconfig -o yaml

K8S Monitoring

Sun, 18 Aug 2024 00:00:00 +0000

Readiness & Liveness

Readiness Probe: This probe determines if a pod is ready to start receiving traffic. Use it to ensure that the pod is fully initialized and ready to handle requests.
Liveness Probe: This probe checks the health of the pod on an ongoing basis. It ensures that the pod is still functioning correctly and can continue to receive traffic. If the liveness probe fails, the pod will be restarted.

Your api should have /health endpoint.

Use readiness to check on things before routing traffic to it for instance. Use liveness to check constantly if you should continue to receive traffic else reschedule.

K8S Networking

Sun, 14 Jul 2024 00:00:00 +0000

CNI plugins

CNI (Container Network Interface) is a standard API which allows different network implementations to plug into Kubernetes. Kubernetes calls the API any time a pod is being created or destroyed. There are two types of CNI plugins:

CNI network plugins: responsible for adding or deleting pods to/from the Kubernetes pod network. This includes creating/deleting each pod’s network interface and connecting/disconnecting it to the rest of the network implementation.
CNI IPAM plugins: responsible for allocating and releasing IP addresses for pods as they are created or deleted. Depending on the plugin, this may include allocating one or more ranges of IP addresses (CIDRs) to each node, or obtaining IP addresses from an underlying public cloud’s network to allocate to pods.

Cloud provider integrations

Kubernetes cloud provider integrations are cloud-specific controllers that can configure the underlying cloud network to help provide Kubernetes networking. Depending on the cloud provider, this could include automatically programming routes into the underlying cloud network so it knows natively how to route pod traffic.

Kubenet

Kubenet is an extremely basic network plugin built into Kubernetes. It does not implement cross-node networking or network policy. It is typically used together with a cloud provider integration that sets up routes in the cloud provider network for communication between nodes, or in single node environments. Kubenet is not compatible with Calico.

K8S Troubleshooting

Sat, 13 Jul 2024 00:00:00 +0000

1

kubectl --v=9 get pods

Authorization / Authentication

Review the API server logs to get more details about the authentication error. This may provide additional clues as to why the request is failing.

Make sure you can reach the Kubernetes API server from your machine:

1

curl -k https://:/healthz

1

curl -k https://:/api/v1/namespaces --header "Authorization: Bearer "

1

kubectl auth can-i --list

1
2
3
4
5


chmod 600 /path/to/admin.crt /path/to/admin.key /path/to/ca.crt

kubectl config view -o jsonpath='{.users}'

curl --cert /path/to/admin.crt --key /path/to/admin.key --cacert /path/to/ca.crt https://:/api/v1/namespaces

K8S Configuration Overview

Mon, 01 Jan 0001 00:00:00 +0000

Initially, Kubernetes was build to manage Docker Containers. Docker is the technology that made containers accessible to the mainstream. Containers were not an entirely new technology. LXC containers existed before and Linux namespaces are a thing since 2002, but Docker made it so easy to use containers. By handling software packaging, distribution and all low level network and device configuration, Docker allowed every developer to start any application in an isolated environment. Promising to finally overcoming the famous: It works on my machine meme. Even the Kubernetes core team admits that Kubernetes would’ve been such a success without Docker.

At some point, a lot of different people and companies tried to solve similar problems regarding containers. They agreed to build some common interfaces for their solutions so that parts of their implementations could be swapped while relying on certain standards. This led to the creation of the Open Container Initiative (OCI), Container Network Interface (CNI), Container Storage Interface (CSI) and Container Runtime Interface (CRI).

Labels

A consistent labeling strategy helps you filter resources quickly and maintain a clear mental map of your cluster.

Labels: Key-value pairs used for grouping and selecting Kubernetes objects. For instance, app=my-app, env=staging, team=payments.
Annotations: Key-value pairs for attaching non-identifying metadata (e.g., version info, contact email, or last-deployed timestamp).

Safe-to-Evict Annotations

Pods marked with safe-to-evict: false cannot be removed or rescheduled during node scaling or bin-packing operations.

Annotation Example:

1
2
3


metadata:
  annotations:
    cluster-autoscaler.kubernetes.io/safe-to-evict: "false"

Naked Pods

Pods created directly without a controller (e.g., Deployments, ReplicaSets) lack the automation needed for rescheduling or scaling.

Naked Pod Example:

1
2
3
4
5
6
7
8


apiVersion: v1
kind: Pod
metadata:
  name: standalone-pod
spec:
  containers:
  - name: app-container
    image: app-image:v1

These pods prevent nodes from being scaled down, leading to idle resources.

Removing Safe-to-Evict Annotations

Safe-to-evict annotations prevent pods from being evicted by the autoscaler. To find pods with this annotation:

1

kubectl get pods --all-namespaces -o json | jq '.items[] | select(.metadata.annotations."cluster-autoscaler.kubernetes.io/safe-to-evict" == "false") | .metadata.name'

Once identified, edit the pod configuration to remove the annotation.

Namespaces

Why Use Namespaces?

Namespaces help:

Isolate Resources: Separate applications and environments (e.g., development, staging, production) within the same cluster.
Manage Access: Apply role-based access control (RBAC) for specific namespaces.
Organize Resources: Group related resources logically for clarity.

Namespace Like Your Life Depends on It

Team-based namespaces: Dev, QA, Prod, or per microservice if that makes sense.
Access Control: Combine namespaces with RBAC (Role-Based Access Control) policies to ensure that only the right people (and services) can mess with your stuff.
Resource Quotas: You can set quotas (e.g., CPU, memory) per namespace, preventing one rogue microservice from hogging all resources.

Take a step back and design your namespace strategy; future you will say thanks.

CRI

With version 1.24, Kubernetes has overcome its dependence on Docker and now only relies on any CRI compliant runtime to start containers. The containerd project was originally developed by Docker Inc and still is the core of Docker, but its ownership was since donated to the CNCF. When a new Pod is scheduled to a node, the kubelet communicates with containerd over the container runtime interface to actually pull the image and start the container with the configuration specified in the pod definition. Containerd is now in charge to set up the Linux namespace, mount volumes or devices and apply seccomp profiles and much more for every new container. That’s a lot of tasks, and not quite according to the UNIX philosophy to do one thing and do it well.

By default, containerd calls runc. The runc binary actually spawns the container process with all its applied properties. It does not pull images, manages the containers file system, sets up networking or monitors the container process. Overly simplified it is just a spawn syscall with a bunch of extra properties for some isolation, but it still runs on the same kernel since containers are NOT VMs.

Master Components

Configuration

Networking

Resources

Resource Requests: This is basically your container’s baseline. If your container requests 200m CPU and 512Mi of memory, the Kubernetes scheduler will place your Pod on a node with at least that much capacity available.
Resource Limits:: This is the upper bound. If your container tries to exceed the limit, it might get throttled (CPU) or even evicted (memory).

Why do we need Requests & Limits?

Ensure fair resource sharing between containers.
Prevent resource hogging by any container.
Help scheduler make informed decisions about where to place pods.
Prevent node crashes from overcommitted memory.

Sample YAML:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


apiVersion: v1
kind: Pod
metadata:
  name: resource-demo
spec:
  containers:
    - name: busybox
      image: busybox
      command: ["sh", "-c", "while true; do echo Hello; sleep 1; done"]
      resources:
        requests:
          memory: "64Mi"
          cpu: "250m"
        limits:
          memory: "128Mi"
          cpu: "500m"

Explanation:

250m means 250 millicores of CPU (¼ core).
The container requests 64Mi memory but can’t exceed 128Mi.

How Scheduling Works with Requests:

Step 1: You apply the pod
Step 2: Scheduler looks for a node with >= requested CPU/memory
Step 3: Pod is scheduled there
Step 4: During runtime, if pod tries to exceed limit:
            - For CPU: throttled
            - For Memory: killed (OOMKilled)

Let’s check out QoS (Quality of Service) Classes as well, how Kubernetes uses requests and limits to categorize pods, and how that affects scheduling and eviction.

Kubernetes QoS Classes

Kubernetes assigns Quality of Service classes to pods based on how you set requests and limits for CPU and memory:

Examples:

Guaranteed Pod (QoS Class: Guaranteed)

1
2
3
4
5
6
7


resources:
  requests:
    memory: "128Mi"
    cpu: "500m"
  limits:
    memory: "128Mi"
    cpu: "500m"

Burstable Pod (QoS Class: Burstable)

1
2
3
4
5
6
7


resources:
  requests:
    memory: "128Mi"
    cpu: "250m"
  limits:
    memory: "256Mi"
    cpu: "500m"

BestEffort Pod (QoS Class: BestEffort)

1

# No resources section

When Node Memory Hits Limit (e.g., High Load)

Eviction Policy Triggered, Kubernetes starts evicting based on QoS priority.

1️. Pod C (BestEffort) → EVICTED first 2️. Pod B or D (Burstable) → May be evicted 3️. Pod A (Guaranteed) → Most protected

Example YAML: Pod with Resource Settings

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


apiVersion: v1
kind: Pod
metadata:
  name: guaranteed-pod
spec:
  containers:
  - name: nginx
    image: nginx
    resources:
      requests:
        memory: "200Mi"
        cpu: "500m"
      limits:
        memory: "200Mi"
        cpu: "500m"

This Pod gets Guaranteed QoS class.

Flow:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


Client applies pod YAML
        │
        ▼
Pod scheduled on node with enough resources
        │
        ▼
Node allocates CPU & Memory based on "requests"
        │
        ▼
If pod exceeds "limits" → It gets throttled or OOMKilled
        │
        ▼
During pressure → Eviction happens in QoS order:
BestEffort → Burstable → Guaranteed

Eviction Priority (Important Concept)

When node runs out of memory, Kubernetes evicts pods based on QoS class:

BestEffort → Burstable → Guaranteed
 (first)                   (last)

So, Guaranteed pods are safest, and BestEffort gets evicted first.

Behavior During Resource Pressure:

Multi-Container Pod Example

Let’s look at a pod with two containers and different QoS setups:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


apiVersion: v1
kind: Pod
metadata:
  name: multi-demo
spec:
  containers:
  - name: app
    image: nginx
    resources:
      requests:
        memory: "100Mi"
        cpu: "200m"
      limits:
        memory: "100Mi"
        cpu: "200m"
  - name: sidecar
    image: busybox
    command: ["sh", "-c", "while true; do echo sidecar; sleep 2; done"]
    resources:
      requests:
        memory: "64Mi"
        cpu: "100m"
      limits:
        memory: "128Mi"
        cpu: "200m"

Resulting QoS for the Pod: Burstable, since sidecar has request ≠ limit

Best Practices

Taints

1
2
3
4
5


➜  kubectl describe node node1 | grep Taints
Taints:             node-role.kubernetes.io/control-plane:NoSchedule

➜  kubectl taint nodes node1 node-role.kubernetes.io/control-plane:NoSchedule-
node/node1 untainted

Security

Secrets

Kubernetes expects the values in stringData to be strings (string)

1
2
3
4
5
6


kind: Secret
metadata:
  name: my-secret
type: Opaque
stringData:
  password: "12345"  # ✅ It is now a string

If you use data, you must ensure that the values are in Base64:

1
2
3
4
5
6
7


apiVersion: v1
kind: Secret
metadata:
  name: my-secret
type: Opaque
data:
  password: MTIzNDU=  # "12345" in Base64

echo -n "12345" | base64

Probes

Probes are how Kubernetes checks if your containers are alive and ready to serve traffic.

They help Kubernetes:

Know if a container needs to be restarted (if it’s unhealthy).
Know if a container is ready to receive traffic.

Three Types of Probes

Liveness Probe

Checks if the container is alive (i.e., not stuck, unresponsive).

If the probe fails, Kubernetes restarts the container.
Useful for apps that might get into a deadlock or stuck state.

Readiness Probe

Checks if the container is ready to accept traffic.

If it fails, traffic is not sent to the container.
Very useful when your app takes time to initialize or depend on something external.

Startup Probe

Used to delay liveness + readiness checks until the app has fully started.

Prevents false failures when app takes longer to start.
Once successful, other probes kick in.

Probe Mechanisms (How They Work) a. Exec Probes

Runs a command inside the container.
If exit code = 0, success; else failure.

Example:

1
2
3
4
5
6
7


livenessProbe:
  exec:
    command:
    - cat
    - /tmp/healthy
  initialDelaySeconds: 5
  periodSeconds: 5

b. HTTP Probes

Makes an HTTP GET request to a specific path/port.
If it returns 2xx or 3xx status code → success.

Example:

1
2
3
4
5
6


readinessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 5

c. TCP Probes

Tries to open a TCP socket to the container on a specific port.
If it connects, it’s considered healthy.

Example:

1
2
3
4
5
6


startupProbe:
  tcpSocket:
    port: 3306
  initialDelaySeconds: 10
  periodSeconds: 10
  failureThreshold: 30

Anatomy of a Probe

Each probe has key fields:

initialDelaySeconds: 10 # wait before first probe
periodSeconds: 5 # how often to probe
timeoutSeconds: 1 # how long to wait for a response
successThreshold: 1 # min consecutive successes
failureThreshold: 3 # after how many failures it’s considered failed

Key Details:

Kubernetes Cluster: Manages the overall orchestration and communicates with the Kubelet on each node.
Kubelet: Responsible for monitoring containers on the nodes, executing the health probes, and reporting back the status to the API server.
Container (Pod): Runs the application, which responds to probe checks like /live, /ready, and /start.
Health Probes: Probes that check container states:
Liveness Probe: Ensures the container is running. If the probe fails, Kubernetes restarts the container.
Readiness Probe: Checks if the container is ready to serve traffic. If it fails, no traffic will be sent to that pod.
Startup Probe: Delays liveness and readiness checks until the application is fully started to avoid premature failure detection.

Complete Example: All Three Probes Together

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


apiVersion: v1
kind: Pod
metadata:
  name: probe-demo
spec:
  containers:
  - name: demo-app
    image: myapp:latest
    ports:
    - containerPort: 8080
    livenessProbe:
      httpGet:
        path: /live
        port: 8080
      initialDelaySeconds: 10
      periodSeconds: 5
    readinessProbe:
      httpGet:
        path: /ready
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 5
    startupProbe:
      tcpSocket:
        port: 8080
      initialDelaySeconds: 0
      periodSeconds: 10
      failureThreshold: 30

Kubernetes on

Kubernetes CPU Throttled

Kubernetes CPU Requests, Limits, and Throttling

CPU Usage vs CPU Throttling

CPU Management in Kubernetes:

Key Terms and Concepts

Task Types in Relation to CPU Access

CPU Requests vs K8s Scheduler

CPU Requests vs Node

CPU Requests vs Limits

CPU Limits

CPU Limits vs Throttling

Backups With Velero

What Do You Back Up In Kubernetes?

etcd Backups In Talos Linux

How To Back Up etcd In Talos

How To Recover etcd In Talos

What’s Velero?

Velero Installation/Setup

Prerequisites

Installing Velero - Client Side

Installing Velero - Server Side

MinIO Setup

Install Using velero Utility

Testing

Manual Backup/Restore

Create A Backup Schedule

Restore From Scheduled Backup

Restores In General

Back Up PVCs Using democratic-csi Snapshots

Kopia

Kubernetes External Services

About External Services

MinIO Example

Kubernetes: Certificates With cert-manager and Let's Encrypt

What is cert-manager?

Installation/Setup

Verify Installation

Getting Certificates From Let’s Encrypt (using Cloudflare DNS)

Getting A Cloudflare API Token

Deploying A cert-manager Issuer - Staging

Request A Real (Staging) Certificate From Let’s Encrypt

Deploying A cert-manager Issuer - Production

Request A Real (Production) Certificate From Let’s Encrypt

Troubleshooting Certificate Requests

What About Ingress?

OpenEBS Replicated Storage Mayastor

Intro and Prerequisites

Dedicated Storage Node

Create New Talos Node(s)

Let’s Verify Our Disk Mount

Worker Nodes Also Need /var/local Mounted

Installing OpenEBS

Privileged Namespace

Helm Installation

Add DiskPool(s)

Troubleshooting

Testing A Replicated PVC

What Is A “Single” Replica Anyway?

Conclusion

Keycloak as an OIDC provider for Kubernetes

Keycloak as an OIDC provider for Kubernetes

Configure a Keycloak client for Kubernetes SSO

Configuring the name and groups claims

Common CLI environment variables

K8S: Configure Kubernetes API Server

K8S: Configure RBAC

Kubernetes Automated Cluster Scaling

Kubernetes Automated Cluster Scaling

Horizontal Pod Autoscaling (HPA)

Vertical Pod Autoscaling (VPA)

Tolerance

Kubernetes Controllers

What is a Replication Controller?

What is a ReplicaSet?

What is a Deployment?

Kubernetes Init Containers

Kubernetes Init Containers

Example: Downloading Configuration from S3

How It Works

Install Using `velero` Utility