# An insight into the essentials: Run Kubernetes yourself ## Why would I do that? I asked Bing Chat: "Is it worth it to run Kubernetes yourself?" It gave me the following answer: *Whether it’s worth running Kubernetes yourself depends on many factors. Here are some pros and cons:* ***Pros:*** * *You have full control over your Kubernetes installation.* * *You can customize your installation to your specific requirements.* * *You can reduce costs by using your own hardware.* ***Cons:*** * *It requires a high level of technical knowledge and experience.* * *It can be time-consuming to run Kubernetes yourself.* * *It can be expensive if you don’t have your own hardware.* *I hope that helps.* What I found very interesting is that **this information is not entirely in line with what the cloud providers advertise**, as they make it seem logical that it is cheaper to run your workload in the cloud. Bing Chats' answer above is a rather objective but simplified list. I think it is a good starting point for this topic. ## Requirements I would like to show you a Kubernetes setup that would fit nicely in **a home lab or a low-demand environment**. For keeping the extent clear I worked with the following circumstances: * A computer with an amd64 architecture, which means any Intel or AMD-based CPU. If you do this on an ARM-based system, let me know in the comments! * A domain that points to the IP address of your internet router or any network gateway that connects your computer to the internet. We need this so the domain can be **secured with Let's encrypt** * Your network router or gateway is set up to forward ports 80 (HTTP) and 443 (HTTPS) to your computer but does not expose any other unnecessary ports or the whole computer for security reasons - remember, it will be available on the internet. * The computer is running Ubuntu with microk8s (tested with v1.27.1) installed, which is a light Kubernetes software from the OS vendor - feel free to **use your own Kubernetes variant, but things might work differently** for you. Below I'll hint at microk8s-specific things. Reference: [Install methods for MicroK8s](https://microk8s.io/docs/install-alternatives) * You are familiar with SSH or have an IDE that connects to the Ubuntu computer. I'm using Visual Studio Code with the Extensions *Remote -SSH* and *Remote Explorer* from Microsoft * You have a deployable Docker image that exposes a webserver port - if not, just stick with the Nginx image from my examples. ## Initial check Reference: [Install methods for MicroK8s](https://microk8s.io/docs/install-alternatives) Let's see if anything is ready: ```bash microk8s status ``` Output: ```bash microk8s is running high-availability: no datastore master nodes: [...] datastore standby nodes: [...] addons: enabled: cert-manager dns ingress [...] disabled: [...] ``` The entry `high-availability` is *no*, but that's okay for now. If you achieve the required redundancy it becomes highly available, as [explained on this docs page](https://microk8s.io/docs/high-availability). But that's not part of this article. What is necessary though are the enabled addons. If any is missing run the command: ```bash microk8s enable dns ingress cert-manager ``` **DNS** is required for Kubernetes Services like **Cert-Manager** to work properly. **Ingress** is required so your Kubernetes Cluster can be accessed from outside, specifically on the HTTP and HTTPS ports. Now let's check if our tool of choice, kubectl is installed: ```bash kubectl ``` This will output: ```bash kubectl controls the Kubernetes cluster manager. # Here appears a list of possible commands. ``` So far so good. ## Step 1: The configuration code ### Kind: Deployment - this is your app The Deployment is a Kubernetes object which builds containers for (but not only) Docker images and puts them in so-called pods. In this example, we use the publicly available Nginx Docker Image. You might want to use your application and at the end of this article, there will be a bonus section on setting up connections to private Docker registries and using images from there. For now, we work with this: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: demo-deployment spec: selector: matchLabels: app: demo replicas: 2 # Amount of parallel running app instances. template: metadata: labels: app: demo # Has to match the selector spec: containers: - name: demo-container image: nginx ports: - containerPort: 80 --- ``` Create a **tutorial-kubernetes-app.yaml** file and paste this snippet in. Watch out for the indentation as YAML files rely on the exact amount of space characters to build the logical hierarchy. Also copy the three dashes `---` at the bottom which is relevant in separating this code block from the next. ### Kind: Service - Exposing the deployment within Kubernetes The service is a Kubernetes object that takes the set of pods from your deployment object and exposes it within Kubernetes. For our example, it might feel like a bit of overkill but let's stick to this abstraction layer as it has a fundamental role to play: Deployments are considered *mortal*, as their number of replications are scaled up and down or pods might become unresponsive or failing. The service finds these pods in the cluster with a predefined selector: ```yaml apiVersion: v1 kind: Service metadata: name: demo-service spec: selector: app: demo ports: - port: 80 --- ``` Note the selector which points to `app: demo` from the deployment. If your container app is running on a different port you can config the last section this way: ```yaml ports: - port: 8080 # whichever port your container exposes targetPort: 80 # we want the default http port ``` This goes into your **tutorial-Kubernetes-app.yaml** as well. Again: also copy the `---`. ### Kind: ClusterIssuer - We are using Let's encrypt The ClusterIssuer is a Kubernetes Object that we will set up in a separate file. Create a file called **issuer.yaml** and paste the following snippet in: ```yaml apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt-staging spec: acme: server: https://acme-staging-v02.api.letsencrypt.org/directory email: YOUR-EMAIL@EXAMPLE.COM privateKeySecretRef: name: letsencrypt-staging solvers: - http01: ingress: class: public # microk8s specific. You might use class: nginx --- apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt-production spec: acme: server: https://acme-v02.api.letsencrypt.org/directory email: YOUR-EMAIL@EXAMPLE.COM privateKeySecretRef: name: letsencrypt-production solvers: - http01: ingress: class: public # microk8s specific. You might use class: nginx ``` So there are **two sets**. One ClusterIssuer gets the name `letsencrypt-staging` the other one `letsencrypt-production` . This is for your sanity because we don't want to run an incomplete configuration against their production server - possibly causing you to run into rate limits. The idea is to work through this example and get it running with the Let's Encrypt Staging Certificate (which the browser prompts as insecure, but that is OK), then switch over to production and get a trusted certificate. In the snippet above look out for the `email:` section and make sure to enter your E-Mail. The ClusterIssuer will send that info over to Let's Encrypt. Also note the Ingress setting `class: public`. This becomes clearer in the next section when we set up the Ingress Object in our microk8s. The rule of thumb is: to pay attention to what kind of Ingress Class your Kubernetes is using and match that class accordingly. > Regarding **tutorial-kubernetes-app.yaml** and **issuer.yaml** if you are wondering how to name these files, you can name them as you fancy, just make sure they are declarative enough so you know the next time you look at it, what it was about :) > > The same applies to entries in `metadata: name:` sections. Kubernetes just needs to refer to the matching string. Name them as you like or better: find a suitable naming convention. ### Kind: Ingress The Ingress object is the connection between your service and the outside. Here the computer will be exposing your app. Copy the following snippet at the end of **tutorial-kubernetes-app.yaml**. ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: demo-ingress annotations: cert-manager.io/cluster-issuer: letsencrypt-staging spec: ingressClassName: public rules: - host: demo.EXAMPLE.COM http: paths: - path: / pathType: Prefix backend: service: name: demo-service port: number: 80 tls: - secretName: letsencrypt-staging hosts: - demo.EXAMPLE.COM --- ``` Exchange the `demo.EXAMPLE.COM` entry with your own domain. As stated in the Requirements section, it has to be the exact name of what you have set up at your domain provider. Note, that the domain appears twice in this snippet: in the `rules:` section and in the `tls:` section. As soon as we apply these code configurations, the cert-manager will look into this definition and requests automatically a certificate for this domain. Also here is the spec entry `ingressClassName: public` which is specific to microk8s. ## Step 2: Applying the code config ### Cert-Manager Have a look at the [cert-manager documentation](https://cert-manager.io/docs/installation/). It does not work right out of the box but needs some configuration to be applied in your Kubernetes cluster. Execute the following in the console: ```bash kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.11.0/cert-manager.yaml ``` This should output quite some feedback: several cert-manager resources are created in your Kubernetes cluster in the namespace `cert-manager`. Now check if all of these are created by running: ```bash kubectl get all --namespace cert-manager ``` This will output something like that: ```plaintext NAME READY STATUS RESTARTS AGE pod/cert-manager-x 1/1 Running 1 19h pod/cert-manager-cainjector-x 1/1 Running 1 19h pod/cert-manager-webhook-x 1/1 Running 1 19h NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/cert-manager ClusterIP removed 9402/TCP 19h service/cert-manager-webhook ClusterIP removed 443/TCP 19h NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/cert-manager-cainjector 1/1 1 1 19h deployment.apps/cert-manager 1/1 1 1 19h deployment.apps/cert-manager-webhook 1/1 1 1 19h NAME DESIRED CURRENT READY AGE replicaset.apps/cert-manager-cainjector-x 1 1 1 19h replicaset.apps/cert-manager-x 1 1 1 19h replicaset.apps/cert-manager-webhook-x 1 1 1 19h ``` This looks good. ### Create the ClusterIssuer Then execute the issuer.yaml This step relies on the cert-manager being configured. ```bash kubectl create -f issuer.yaml ``` Note: `kubectl create` commands can be executed only once and then the resources are already created. The next time it will throw an error, that it already exists. If you want to roll back your changes run: `kubectl delete -f issuer.yaml` and then you can recreate. Pay attention to the **metadata names** of the elements you are creating and deleting. Now check if the ClusterIssuer is there: ```bash kubectl get clusterissuer -o wide ``` The ideal output looks something like this (shortened): ```plaintext NAME READY STATUS letsencrypt-staging True The ACME account was registered ... letsencrypt-production True The ACME account was registered ... ``` **If there is an error in the status block** you have to check what exactly fails. Highly possible are networking issues, e.g. the API of letsencrypt.org can't be contacted. In that case, check if the microk8s DNS addon is set up. ### Apply the app configuration Now comes the part we have long been working towards. Applying the configuration: ```bash kubectl apply -f tutorial-kubernetes-app.yaml ``` The output: ```plaintext deployment.apps/demo-deployment created service/demo-service created ingress.networking.k8s.io/demo-ingress created ``` Perfect! If not, no worries: there is a big chance that kubectl will have some remarks. Let me take you to the exciting journey of two issues I mostly encountered: ### Possible issue 1: Invalid config ```plaintext The request is invalid: patch: Invalid value: "map[metadata:map[annotations:map[kubectl.kubernetes.io/last-applied-configuration:{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{},\"name\":\"demo-service\",\"namespace\":\"default\"},\"spec\":{\"port\":80,\"ports\":null,\"selector\":{\"app\":\"demo\"}}}\n]] spec:map[port:80 ports:]]": strict decoding error: unknown field "spec.port" ``` To understand whats happening, kubectl takes your file, parses the yaml syntax and applies it to Kubernetes. If parsing is ok it but the config does not match the Kubernetes API the output can be quite cryptic. Above, it finds an "unknown field 'spec.port'", which is nonchalantly expressed that `port: 80` is wrongly indented and not in a map or array format. It most probably does not belong to `spec:` but to its child element `ports:` ### Possible issue 2: Pods not loading Sometimes the pods don't get started. You can check that with: ```bash kubectl get pod --namespace default ``` If one of the pods is in `STATUS: pending` that means it failed to start. Reasons for that are as simple as that the container could not be loaded or that a dependency is missing (see below, Private Registry). You can find some more evidence of where the pod stopped by looking at the exact name in the list and using it in the following command: ```bash kubectl describe pod PODNAME --namespace default ``` ## Step 3: It runs (kind of) ![Image with two nginx responses: 404 is not correct and should be fixed in the config, the default "Welcome to nginx" page should appear.](https://cdn.hashnode.com/res/hashnode/image/upload/v1684335320434/6a374ff3-1749-4634-b83d-874596de6356.jpeg align="center") *If you see a 404 there is something not correct and should be fixed in the config. The default "Welcome to nginx" page should appear with the docker image used in the example.* Open your browser and enter the domain name you configured. Make sure you are accessing with https. As soon as you hit the website you should be prompted with an insecure certificate warning. Since it is your website you can proceed to the page and check the certificate. There are two options: * The certificate is from Kubernetes itself and has the name "**Kubernetes Ingress FAKE Certificate**". In that case, your **Let's Encrypt config is not in place** and the Ingress Object serves with this self-signed certificate. * The certificate is from Let's Encrypt and is something like: "(STAGING) Artifical Apricot R3" from Organization "(STAGING) Let's Encrypt". That's what we want. If it's the second option, you can proceed by exchanging the `letsencrypt-staging` entries in your Ingress definition with `letsencrypt-production` . It should then look like this: ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: demo-ingress annotations: cert-manager.io/cluster-issuer: letsencrypt-production spec: ingressClassName: public rules: - host: demo.EXAMPLE.COM http: paths: - path: / pathType: Prefix backend: service: name: demo-service port: number: 80 tls: - secretName: letsencrypt-production hosts: - demo.EXAMPLE.COM --- ``` Then execute your yaml config again: ```bash kubectl apply -f tutorial-kubernetes-app.yaml ``` Now, that should do the trick! The output should contain info that states: `demo-ingress configured`. Double-check that by running: ```bash kubectl describe ingress ``` That should output a section which looks similar to this (shortened output): ```bash Annotations: cert-manager.io/cluster-issuer: letsencrypt-production [...] CreateCertificate cert-manager-ingress-shim Successfully created Certificate "letsencrypt-production" DeleteCertificate cert-manager-ingress-shim Successfully deleted unrequired Certificate "letsencrypt-staging" ``` The next time you clear the browser cache or open the website from a private browsing window, hitting https://demo.example.com should give you a website with a valid certificate from organisation "Let's Encrypt" ## Step 4: Using a Docker image from a private registry *For this step, you need Docker installed on the Kubernetes host machine or alternatively Docker installed on another machine and be able to export the* `.docker/config.json` Most probably you will not be satisfied by having an nginx dummy page deployed on your TLS-secured domain at this point. So let's say you have already a dockerized app image ready at your private registry. This is where we have to make a guess: the registry is **private-registry.example.com/demo-containers/** and your image is to be found at **private-registry.example.com/demo-containers/demo:latest** > The following section derives from the official documentation: [Pull an Image from a Private Registry | Kubernetes](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/) > > I'll keep it a bit shorter and related to the example. Please refer to the docs. Since the registry is private, Kubernetes will have to do an authentication for retrieving the image. That's where Secret Objects become relevant. First, you need to do a Docker login. This depends on how your registry is set up but likely looks like this: `docker login private-registry.example.com -u USERNAME --password-stdin <<< "$REGISTRY_SECRET_KEY"` When Docker logged in successfully, it will put an auth token in your Docker config.json. Check it with the following command: ```bash cat ~/.docker/config.json ``` If you find the name of the registry and an auth token within that file, you will be able to add it as a secret to Kubernetes. So replace the `PATH_TO_HOME` part and execute the following script: ```bash kubectl create secret generic regcred --from-file=.dockerconfigjson=PATH_TO_HOME/.docker/config.json --type=kubernetes.io/dockerconfigjson ``` The output: ```bash secret/regcred created ``` That is the name we gave the secret: `regcred` and that's what you have to add to the Deployment Object. Update your yaml code: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: demo-deployment spec: selector: matchLabels: app: demo replicas: 1 # tells deployment to run 2 pods matching the template template: metadata: labels: app: demo spec: containers: - name: demo-container # image: nginx image: private-registry.example.com/demo-containers/demo:latest ports: - containerPort: 80 imagePullSecrets: - name: regcred --- ``` Then execute your yaml config again: ```bash kubectl apply -f tutorial-kubernetes-app.yaml ``` And to make sure your image is being correctly pulled from your private registry check the pod: ```bash kubectl describe pod --namespace default ``` Chances are your pod(s) will be stuck in `Status: Pending`, then have a look at the Event log at the end of the output - there might be an error message waiting for you. But if your Pods will be in `Status: Running` then check your browser - time to get excited :) ## Final remarks Microk8s is production-grade Kubernetes but is not perfect out of the box. If you want to run an app with it read more about how to configure it. For that, you might be interested in the [MicroK8s How To guides](https://microk8s.io/docs/how-to). How did you get along? Could you follow or got stuck? If you made it to the end you have quite something achieved! I'd be happy to hear from you!