# Certified Kubernetes Admin # Section 2: Core Concepts ## Cluster Architecture - Master - Manage, Plan, Schedule & Monitor - Kube API Server, manages cluster operations - Kube Controller Manager - ETCD, cluster info - kube-scheduler, schedules container applications on nodes. - Worker Node - Runs containers - Kublet - Is what interacts as a client to the Master/API server. These are what perform commands on behalf of the Kube API Server & manages the containers on the nodes. - Kube Proxy - Helps support network connectivity between different containers running on different worker nodes. ## Docker vs. ContainerD - Docker was one of the original container services. - ContainerD was a component of Docker. - K8 was developed to orchestrate Docker primarily. - Now supports other container runtimes. - Implemented CRI or Container Runtime Interface. - ContainerD - Is separate from Docker now. - `ctr` commands, limited in usage/functionality. - debugging tool - for containerd runtime - `nerctl` docker like cli & provides greater features/functionality. - general purpose cli - for containerd runtime - `crictl` cli for CRI compatible containers - supports various runtimes. - not recommended for prod use. - debugging tool - manually setting CRI endpoints, `--runtime-endpoints` ## ETCD For Beginners - Port 2379 - `etcdctl` cli for service interaction - `etcdctlset set key1 value1` - create key-value pair - `etcdctlset get key1` - get key-value pair - `--version` to obtain API version in use. - Set via `ETCD_API=$VERSION` env variable. - `version` for v3 & `--version` for v2 ETCD releases respectively. ## ETCD in Kubernetes - Supports manual or `kubeadm` setup/configuration. - `kubeadm` set it up as its own pod in `kube-system` namespace. - use `kubectl` command to interact with it. - HA option `--initial-cluster` option. - Reference slide on: **(Optional) Additional information about ETCDCTL Utility** for info on: - TLS certs - Difference in commands between versions. ## Kube-API Server - Primary mgmt component. - Is the service that `kubectl` interacts with. - Uses ETCD as data store. - Manual HTTP methods can be used as well. - When a request is made the API server then sends the request to: - `kube-scheduler` that then sends the request to the appropriate `kublet` worker node. - After `kublet` completes, data gets updated in ETCD. - Using `kubeadm` it will deploy it as a pod, e.g. `kube-apiserver-master`. ## Kube Controller Manager - Node Controller - Verifies & ensures state of pods - Performs remediation - Interacts with Kube API Server - Replication Controller - Monitors replica sets & pods - Numerous controllers performing various aspects of the K8 service - Using `kubeadm` it will deploy it as a pod in `kube-system` namespace, e.g. `kube-controller-manager-master`. ## Kube Scheduler - Schedules PODs on nodes. - which POD goes on which node. - Scheduling process. 1. Filter which nodes can accommodate the request. 2. Rank the nodes that are available to support the request based on requirements. ## Kublet - Service that runs on the node to deploy containers. - Primary service that interacts with the Kube Controller Master service. - Takes requests from the Kube Scheduler. - Registers node to K8 cluster. - `kubeadm` **does not** deploy Kublets. - Deploying Kublets is done manually on the worker nodes. ## Kube Proxy - Management of POD networks - A Kube service is created with addresses that can be static for when the underlying PODs for a Kube service change. - e.g. A POD may have multiple containers in a POD with various address & this service allows for connectivity between them. - Is a virtual component. - Runs on each node. - Creates the appropriate rules on the nodes to allow access to whatever Kube service. ## PODs - A application is developed to run in a container. - Containers are encapsulated in a POD. - A POD is a single instance of a application. - You don't add an additional container to a POD to scale. - You create a separate POD on same or different node. ## Multi-Container PODs - PODs usually have 1:1 containers to a POD. - A POD can have multiple containers to a POD but not the same type of container. - Helper or supporting containers, (e.g. for additional processing, etc) - PODs take care of the overhead of manually: - Creating new containers for scaling. - Liking of helper or support containers. - Management of network connectivity between containers. - Management of volumes. - Management of the state of the application for HA & scaling. ## kubectl - Example of creating a simple `nginx` POD. - By default this will pull the image from the public DockerHub repository. ``` # kubectl run nginx --image nginx ``` - List the current running PODs. ``` # kubectl get pods ``` - By default This doesn't allow for end-user connectivity but you can use internal connectivity for communication. ## PODs with YAML - K8 uses YAML files for input for cluster management. - To create objects within the cluster, PODs, networks, volumes, scaling, etc - The file format requires 4 top-level specific keywords. - `apiVersion` - The K8 API version being used. - `kind` - The type of object to create. - e.g. POD, Service, ReplicaSet, Deployment - `metadata` - Custom metadata for your needs or uses. - `spec` - The containers to use for the POD. - Example of creating a POD from a file. ``` # kubectl create -f pod-definition.yml ``` - Example of pod-definition YAML file: ```yaml --- apiVersion: v1 kind: Pod metadata: # metadata is optional. name: myapp labels: app: myapp type: frontend spec: containers: - name: redis - image: redis123 ``` - To see detailed info of a POD. ``` # kubectl describe pod myapp-pod ``` ## DEMO - PODs with YAML - Example of creating of POD using YAML file. - `apply` can be used in place of `create` in this instance. ``` # kubectl apply -f pod.yaml ``` ### Practice Test - PODs - practice tests are on 3rd party due to Udemy limitations - https://uklabs.kodekloud.com/topic/practice-test-pods-2/ - https://learn.kodekloud.com/user/courses/udemy-labs-certified-kubernetes-administrator-with-practice-tests ## Practice Commands - Create new pod with specified image. ``` # kubectl run $POD_NAME --image $IMAGE ``` - Delete pod. ``` # kubectl delete $POD_NAME ``` - Create a pod using definition file. Example file contents, `pod-sample.yaml`: ```yaml apiVersion: v1 kind: Pod metadata: name: redis spec: containers: - name: redis image: redis ``` - Create the pod from the definition pod file. - Note: A pod can be reconfigured by editing the file & then re-running this command. ``` kubectl apply -f pod-sample.yaml ``` - SIMPLER create a pod yaml file automagically from `kubectl` output. ``` kubectl run $POD --image $IMAGE --dry-run=client -o yaml > $FILE.yaml ``` - A pod can be edited with: ``` kubectl edit $POD ``` ## Recap - ReplicaSets ### Replication Controller - Provides HA. - Helps run multiple instances of a single POD. - In single POD instances, it can rebuild the POD if it fails. - Keeps specified number of PODs running. - Spans multiple nodes in the cluster. - Helps provide Load Balancing & Scaling - This is the legacy implementation of this component/service. - Is replace by Replica Set ### Replica Set - Modern replication component/service. - Provides same functionality as Replication Controller. - Differences in implementation such as: - TBD ### Defining Replication Controller - Similar to creating a POD definition file. - The `kind` is `ReplicationController` - The `spec` area is important to specify the `- template` that will be used. - A template is merely a POD definition without the `apiVersion` or `kind` options set/defined. - Example `ReplicaSet` ```yaml apiVersion: v1 kind: ReplicationSet metadata: name: myapp-rc labels: app: myapp type: front-end spec: template: metadata: name: myapp-pod labels: app: myapp type: front-end spec: containers: - name: nginx-container image: nginx replicas: 3 ``` - Note there are 2 `spec` options. One for the `- template` & the other for the `containers`. - The option `replicas` is what defines how many PODs in this ReplicaSet to run. - You will create the ReplicaSet as you would with a normal POD definition file. ``` kubectl create -f $replica-set.yml ``` - To review Replication Controllers: ``` kubectl get replicationcontroller ``` - When reviewing the PODs via `kubectl get pods` the PODs will have the name of the Replication Controller pre-pended to the POD names. ``` example goes here. ``` ### Defining Replica Set - Uses slightly different `apiVersion`, `apps/v1` - Uses a `selector` option - Must be set & uses the `matchLabels` option. - This can match any `label` set in the `metadata` area of the `spec` section. - In this example, `type: front-end` is the label we're using. ```yaml apiVersion: apps/v1 kind: ReplicaSet metadata: name: myapp-replicaset labels: app: myapp type: front-end spec: template: metadata: name: myapp-pod labels: app: myapp type: front-end spec: containers: - name: nginx-container image: nginx replicas: 3 selector: matchLabels: type: front-end ``` - To review Replica Sets: ``` kubectl get replicaset ``` - When reviewing the PODs via `kubectl get pods` the PODs will have the name of the ReplicaSet pre-pended to the POD names. ### Labels & Selectors - The role of the Replica Set is to monitor the POD & deploy new ones upon any POD failures. - The **selectors** use the **labels** to identify which of the PODs that are related to the Replica Set to manage/monitor. - If existing PODs exist that match the defined **selector** it will begin to manage them and not create new PODs for the Replica Set. - The **spec** option is required for the Replica Set even if we're just using it to manage existing PODs. - This is because if the existing PODs fail, it will need to know which **template** to use for deploying new PODs. ### Scaling - There are a few methods to scale the Replica Set. - Method 1 - Updating the **replicas** option in the definition file & use the **replace** command to scale the PODs in the Replica Set, e.g. change `replicas` from 3 to 6 then execute command. ``` kubectl replace -f $replica-set.yml ``` - Method 2 - Use the `kubectl scale` command with definition file or specify the Replica Set name. - Note: When using this method with the definition file, the file **is not** updated with the new value. Thus if you want this to be "permanent", update the definition file itself. ``` kubectl scale --replicas=6 -f $replica-set.yml kubectl scale --replicas=6 replicaset myapp-replicaset ``` ### Commands - Delete Replica Set, this removes underling PODs as well. ``` kubectl delete replicaset $myapp-replicaset ``` - Edit running Replica Set ``` kubectl edit replicaset $replica-set ``` # Deployments - Is a object that encapsulates **ReplicaSets** & **PODs**. - Functionality includes updating PODs, rolling updates, undo changes, pausing & resuming changes. - The deployment file is basically the same as a `ReplicaSet`, with the **kind** option being set to `Deployment`. - List deployments: ``` kubectl get deployments ``` - See all related objects: ``` kubectl get all ``` ## Certification Tip - kubectl conventions - https://kubernetes.io/docs/reference/kubectl/conventions/ - The `kubectl run` command can make creating YAML files simpler. - The command can also be used to quickly save time instead of manually performing steps individually. - Example - Create NGINX POD: ``` kubectl run nginx --image=nginx ``` - Example - Generate POD manifest file: ``` kubectl run nginx --image=nginx --dry-run=client -o yaml > nginx-pod.yaml ``` - Example - Create a deployment: ``` kubectl create deployment nginx --image=nginx ``` - Example - Generate deployment manifest file: ``` kubectl create deployment nginx --image=nginx --dry-run=client -o yaml nginx-deployment.yaml ``` - Example - Method A - Update deployment after making changes to manifest file: ``` kubectl create -f nginx-deployment.yaml ``` - Example - Method B - Update deployment after making changes to manifest file (v1.19+): ``` kubectl create deployment --image=nginx nginx --replicas=4 --dry-run=client -o yaml > nginx-deployment.yaml ``` # Services ## NodePort - NodePort Service - Listens on a port on a node & forwards that traffic to the specified POD. - 3 ports used. - TargetPort on the POD. - Port on the Service itself that supports the cluster connectivity. - This uses a ClusterIP internally. - Will default to the NodePort if not defined. - NodePort is the port on the node itself that end-users will connect to. - This will forward traffic to the TargetPort thru the Service Port. - This uses ports 32000-32767 by default. - Will use available port within range if not defined. - Uses Random Algorithm & Session Affinity for traffic management between PODs. - Thus acts as a sort of load balancer/traffic manager. - Supports being spread across multiple nodes & pods. - Definition file uses the following: ```yaml apiVersion: v1 kind: Service metadata: name: myapp-service spec: type: NodePort ports: - targetPort: 80 # Port on POD. port: 80 # Service port used internally within the cluster. nodePort: 30008 # Port that is exposed to end-users. selector: # The labels used to associate with the POD the Service supports. app: myapp type: front-end ``` - Create a Service from definition file: ``` kubectl create -f service-definition.yaml ``` - List Services: ``` kubectl list services ``` ## ClusterIP - ClusterIP Service - VIP within cluster, for inter-cluster communication. - Can be connected by IP or the Service name within the cluster. - ClusterIP definition file: ```yaml apiVersion: v1 kind: Service metadata: name: back-end spec: type: ClusterIP ports: - targetPort: 80 # Port on POD. port: 80 # Service port used internally within the cluster. selector: # The labels used to associate with the POD the Service supports. app: myapp type: back-end ``` - Create a ClusterIP from definition file: ``` kubectl create -f service-clusterip.yaml ``` ## LoadBalancer - LoadBalancer Service - Deploys a load balancer within supported cloud environments for HA. ```yaml apiVersion: v1 kind: Service metadata: name: myapp-service spec: type: LoadBalancer ports: - targetPort: 80 # Port on POD. port: 80 # Service port used internally within the cluster. nodePort: 30008 selector: # The labels used to associate with the POD the Service supports. app: myapp type: back-end ``` # Namespaces - Namespaces are akin to resource fencing. - Isolation - They can be isolated from each other. - Policies - who can do what. - Resource Limits or quotas. - The Namespaces included with K8 are: - kube-system - used for internal system management. - default - for normal day-to-day usage. - kube-public - used for all users(?). - Namespaces can communicate with each other using the FQDN of another Namespace. - DNS can look similar to the following, `db-service.dev.svc.cluster.local` - `cluster.local` is the top level domain. - this would be your domain, e.g. `example.com` - `svc` is the subdomain indicating a service. - `dev` indicates the namespace we're working in. - `db-service` is the particular `svc`/service in the `dev` namespace. ### Namespace Commands - Obtain PODs in another namespace using, `--namespace=`: ``` kubectl get pods --namespace=kube-system ``` - Create a POD in a specific namespace: ``` kubectl create -f pod-definition.yaml --namespace=dev ``` - The namespace can be added to the POD definition file. ``` apiVersion: v1 kind: Pod metadata: name: myapp-pod namespace: dev # define the namespace here! labels: app: myapp type: front-end spec: containers: - name: nginx-container image: nginx ``` - Namespace definition file example: ```yaml apiVersion: v1 kind: Namespace metadata: name: dev ``` - Creating namespace using definition file: ``` kubectl create -f namespace-dev.yaml ``` - Manually creating namespace via `kubectl`: ``` kubectl create namespace dev ``` - Switch namespaces: ``` kubectl config set-config $(kubectl config current-context) --namespace=dev ``` - See PODs in all namespaces: ``` kubectl get pods --all-namespaces ``` ## Resource Quotas - Resource quota definition file: ```yaml apiVersion: v1 kind: ResourceQuota metadata: name: compute-quota namespace: dev spec: # define limits here. hard: pods: "10" requests.cpu: "4" requests.memory: 5Gi limits.cpu: "10" limits.memory: 10Gi ``` - Create Resource Quota from def. file: ``` kubectl create -f compute-quota.yaml ``` # Imperative vs Declarative - Imperative - To be precise & state each step, (explicit in Pythonic terms) - example, manually deploying NGINX & web app, step by step. - think bash commands. - In K8 terms, it would be manually executing each `kubectl` command. - Create & update objects - Declarative - To say what you would like & allow the app to execute it as it sees fit. - example, requesting for NGINX VM, running on port 8080, download app. - think ansible playbooks. - In K8 terms, it would be using definition files to deploy the app. - `kubectl apply -f deployment.yaml` - Supports single file manifests. - `kubectl apply -f /path/to/deployment_files` - Supports reading multiple manifest files. - The `apply` sub-command can be used for creating & updating objects. ## Certification Tips - Imperative Commands #### POD - Create an NGINX Pod ``` kubectl run nginx --image=nginx ``` - Generate POD Manifest YAML file (-o yaml). Don't create it(--dry-run) ``` kubectl run nginx --image=nginx --dry-run=client -o yaml ``` #### Deployment - Create a deployment ``` kubectl create deployment --image=nginx nginx ``` - Generate Deployment YAML file (-o yaml). Don't create it(--dry-run) ``` kubectl create deployment --image=nginx nginx --dry-run=client -o yaml ``` - Generate Deployment with 4 Replicas ``` kubectl create deployment nginx --image=nginx --replicas=4 ``` - You can also scale a deployment using the kubectl scale command. ``` kubectl scale deployment nginx --replicas=4 ``` - Another way to do this is to save the YAML definition to a file and modify ``` kubectl create deployment nginx --image=nginx --dry-run=client -o yaml > nginx-deployment.yaml ``` #### Service Create a Service named redis-service of type ClusterIP to expose pod redis on port 6379 ``` kubectl expose pod redis --port=6379 --name redis-service --dry-run=client -o yaml ``` - (This will automatically use the pod's labels as selectors) - Or ``` kubectl create service clusterip redis --tcp=6379:6379 --dry-run=client -o yaml ``` - This will not use the pods labels as selectors, instead it will assume selectors as app=redis. You cannot pass in selectors as an option. So it does not work very well if your pod has a different label set. So generate the file and modify the selectors before creating the service. - Create a Service named nginx of type NodePort to expose pod nginx's port 80 on port 30080 on the nodes: ``` kubectl expose pod nginx --type=NodePort --port=80 --name=nginx-service --dry-run=client -o yaml ``` - This will automatically use the pod's labels as selectors, but you cannot specify the node port. You have to generate a definition file and then add the node port in manually before creating the service with the pod. - Or ``` kubectl create service nodeport nginx --tcp=80:80 --node-port=30080 --dry-run=client -o yaml ``` - Create a POD & ClusterIP service together. ``` kubectl run httpd --image=httpd:alpine --port=80 --export=true ``` #### Reference: - https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands - https://kubernetes.io/docs/reference/kubectl/conventions/ # Kubectl Apply Command - Declarative use in creating & updating objects. - Internal process of change management: - Local file -> Last Applied Config -> Kubernetes Live Config - The YAML file is converted into a JSON file. - Changes are identified between all of the objects, (file, applied config & live config) - Once these changes are identified, they will get applied based on the changes needing to be applied. - Last applied configuration is stored in the `annotations` field in Live Config. # Section 3: Scheduling ## Manual Scheduling - Uses the `nodeName` property in the **spec** area to identify which node to deploy instances to. ``` spec: containers: - name: nginx image: nginx ports: - containerPort: 8080 nodeName: $NODE ``` - The node can only be defined during creation. - You can use the **Binding** object to attach to an existing POD. - Must convert to JSON format & send to Binding API manually. - To see if the scheduler component exists on the system: ``` kubectl get pods -n kube-system ``` - To recreate a POD use the **replace** option: ``` kubectl replace --force -f nginx.yaml ``` - Monitor status of command using built in **--watch** argument: ``` kubectl get pods --watch ``` - Identify which node the POD is running on: ``` kubectl get pods -o wide ``` ## Labels & Selectors - Standard method to group objects together. - Labels are properties attached to each item. - Selectors are filters of these properties. - Labels are utilized in the **metadata** section of the POD definition. - Label selection: ``` kubectl get pods --selector app=App1 ``` - Note: The label properties can be found in both the **ReplicaSet** metadata area as well as the **template** metadata area. - The template metadata area is for the POD. - The ReplicaSet metadata area is for the ReplicaSet. - The Selector object will match resource objects, e.g. PODs to ReplicaSets ``` apiVersion: apps/v1 kind: ReplicaSet metadata: name: simple-webapp labels: app: App1 function: Front-end spec: replicas: 3 selector: matchLabels: app: App1 template: metadata: labels: app: App1 function: Front-end spec: containers: - name: simple-webapp image: simple-webapp ``` ## Annotations - Are custom data points such as: - email contact - build version - any piece of data that can be used for integration purposes ### Certification Tips - Commands 2 - List all objects & labels without headers: ``` kubectl get all --show-labels --no-headers ``` ## Taints & Tolerances - POD to Node relationships. - Are used to set restrictions for PODs to be deployed on Nodes. - Taints say what can NOT be allowed on a Node. - Taints on Nodes. - You taint a node. - Tolerances say which POD CAN be allowed on a Node. - Tolerances on PODs. - A POD has tolerances. ## Taints - Apply a taint to a node: ``` kubectl taint nodes $node-name $key=$value:$taint-effect ``` - Where the `$key:$value` would be `tier=frontend` - The `$taint-effect` would be on of: - **NoSchedule** - PODs will not be scheduled on the node. - **PreferNoSchedule** - Try to avoid placing on the node. - **NoExecute** - New PODs will not be deployed to this & existing PODs that do not **tolerate** the **taint** will be evicted/removed from the Node. ## Tolerances - Set a tolerance in a POD definition file: ``` apiVersion: v1 kind: Pod metadata: name: simple-webapp spec: containers: - name: simple-webapp image: simple-webapp tolerations: - key: "app" operator: "Equal" value: "blue" effect: "NoSchedule" ``` ## Taints - NoExecute - Taints don't tell a POD to go to a specific Node. - It merely will ensure that PODs deployed on it meet the specific criteria. - Meaning that while a taint could be applied to a node & a POD be tolerable of that taint, the POD could still be deployed to a Node that DOES NOT have a taint applied. - Example: You have 3 nodes, one with a blue taint & a POD (D) that is tolerable of that taint. POD (D) upon deployment could go to that Node or another node that is available without that or any taint applied to it. - Obtain taints of the Kubernetes master node: ``` kubectl describe node kubemaster | grep Taints ``` - Show events for a POD: ``` kubectl describe pod $POD # then view the Events area ``` - To remove a taint, add a `-` to the end of the taint: ``` kubectl tiant nodes $nodes $key=$value:$taint-effect- ``` ## Node Selectors - A new property of `nodeSelector` is available to add to the `spec` section of the POD definition file. - Selectors are labels in key-value format that are applied to nodes. - Nodes must be labeled prior to POD creation in order to work. ## Label Nodes - Labeling nodes: ``` kubectl label nodes $node $label-key:$label-value ``` - Labels are simple and don't offer functionality to exclude or include additional labels. - e.g. Specifying a POD to use anything but small or just medium/large wouldn't be possible with labels alone. ## Node Affinity - Primary purpose is to ensure that specific PODs are deployed to specific nodes for workload needs/requirement. - Reference docs for further info. - https://kubernetes.io/docs/tasks/configure-pod-container/assign-pods-nodes-using-node-affinity/ - You will literally need to use the docs for obtaining examples of affinities. Otherwise it'll be too much to try to recall. - There are 2 nod affinity types currently. - `requiredDuringSchedulingIgnoredDuringExecution`: The scheduler can't schedule the Pod unless the rule is met. This functions like `nodeSelector`, but with a more expressive syntax. - `preferredDuringSchedulingIgnoredDuringExecution`:  The scheduler tries to find a node that meets the rule. If a matching node is not available, the scheduler still schedules the Pod. ## Node Affinity vs Taints & Tolerances - With Taints & Tolerances a specific POD could still be deployed on a Node that does not have the specific taint applied. - This is how this particular function operates. - With Node Affinity you may be able to ensure that specific PODs are deployed to specific Nodes but other PODs may be able to be deployed to that Node. - This is the reverse problem of Taints & Tolerances. - Thus a combination of them both is required. - First we taint the nodes & then we set the affinity to the PODs. - This will ensure that the PODs are deployed as per our requirements. ## Resource Requirements & Limits - The `resources` property is used within the `spec` area for the `container`. - This uses the `requests` sub-property to define the needed resources for the POD. - `memory` for memory. - `cpu` for processing. - Note the differences between "G/M/K" vs "Gi/Mi/Ki" for resource measurements. ### Resource Limits - This uses the `limits` sub-property to define the resource limits for the POD. - If a POD exceeds limit for memory it will be terminated & OOM'd. - Out Of Memory - If a POD attempts to exceed CPU resources it will be throttled. - By default there are no limitations for the PODs. - You want to ensure to have Request set for PODs to ensure proper resource allocation. ### Limit Ranges - Used to define default resources for PODs. - Kind `LimitRange` - Affects only newly created PODs. ## Resource Quotas - Used to set hard limits for resources. - Kind `ResourceQuota` ## POD Editing - Extract the pod definition in YAML format to a file using the command: ``` kubectl get pod webapp -o yaml > my-new-pod.yaml ``` - These are the available editable options for POD specifications: - `spec.containers[*].image:` - `spec.initContainers[*].image:` - `spec.activeDeadlineSeconds:` - `spec.tolerations` - Editing certain values will have it save to a file in `tmp` dir. - Deployments any field/property can be edited. ## Daemon Sets - Daemon Sets are similar to ReplicaSets in it helps deploy multiple instances to PODs. - It runs one copy of POD on each Node. - It will add/remove a POD when a Node is added/removed. - Ensures that one copy of the POD is present in all Nodes of the cluster. - One use case is a monitoring agent used to run on every Node in the cluster. - Another is the kube-proxy service/agent or any agent based component. - A Daemon Set will ensure this application in the POD is running on every Node. - The Daemon Set definition file is similar to the ReplicaSet definition file in terms of structure. - Get running Daemon Sets ``` kubectl get daemonsets -A ``` - How Daemon Sets work: - TBD - Simple DaemonSet definition creation using the `deployment` option with a dry-run. - Then update the output file to: - Remove replicas, strategy, & status fields. - Remove RSS, mnemonic - Update the type from `deployment` to `daemonset` ``` kubectl create deployment $deployent -n $namespace --image=$image --dry-run=client -o yaml $daemonset.yaml ``` - Shorthand for `daemonsets` is `ds` ## Static PODs - If you don't have an actual cluster implemented, you can manually deploy static PODs. - k8s can read files from the directory `/etc/kubernetes/manifests`. - Manifests in this directory will be automatically deployed. - Updates to the manifests in this directory will automatically be applied & PODs redeployed to reflect these changes. - This is only for PODs, other services are not supported. - Remember the `kubelet` works at the POD level. - The directory is set via the `--pod-manifest-path=` option in the service file. - This directory can be set in the config file, `kubeconfig.yaml`, `staticPodPath`. - Static PODs are managed via the container service cli since there is no service for the `kubectl` command to interact with. - Where the container service is one of: - `docker` for docker - `crictl` for cri-o - `nerdctl` for containerd - Static PODs are still registered to the kubeapi service. - Static PODs can't be managed via this service though. - Use cases: - Creating the k8s cluster PODs. - The internal k8s already does this. - Static PODs vs Daemon Sets - Daemon Sets are API objects managed by the cluster - Static PODs are managed by the `kubelet` locally. - Daemon Sets are suited for agents on nodes themselves. - Static PODs are suited for the underlying POD cluster deployment. - Both are ignored by the kube-scheduler. - A simple way to identify if something is not static is by the name. - If it has random characters, thats a non-static POD. - FOLLOW-UP: - The identifying of PODs was somewhat complex for the final issue. - It required logging into the `node01` & removing the POD manifest file. - Remember the node name is appended to the POD. - e.g. $POD-node01 or $POD-controlplane - Use the `kubectl get` command to retrieve the "ownerReference" of a POD. ``` kubectl get pods $POD -n $namespace -o yaml ``` ## Multiple Schedulers - The default scheduler is called, "default-scheduler". - Configuration is defined in `scheduler.yaml` - A `command` object example: ``` # rest of spec file spec: containers: - command: - kube-scheduler - --address=127.0.0.1 - --kubeconfig= - --config= ``` - A different scheduler can be defined in the POD manifest file using the `schedulerName` object in the `spec` area of the manifest. ## Scheduler Profiles - The `priorityClassName` provides the ability to define what PODs have greater priority for deployment in the scheduling queue. - Scheduling -> filtering -> scoring -> binding # Section 4: Logging & Monitoring ### Monitor Cluster Components - Metrics Server is a simple built in metrics provider. #k8s-monitoring - View resource metrics ``` kubectl top node kubekctl top pod ``` - Deploy metrics service, download component. ``` git clone https://github.com/kodekloudhub/kubernetes-metrics-server.git ``` - Deploy metrics service, enable via `kubectl`, in git repo directory. ``` kubectl create -f . ``` ### Application Logs - Introducing the `logs` sub-command, this will return application logging of a POD. ``` kubectl logs -f $POD ``` - If there are multiple containers in a POD, you will need to specify the container you're wanting logs for. ``` kubectl logs -f $POD $CONTAINER ``` - This is a simple implementation of logging for containers/PODs. - Logs can be parsed using standard/common methods. ``` kubectl logs pods $POD | less kubectl logs pods $POD | grep $ERROR ``` # Section 5: Application Lifecycle Management ## Rolling Updates & Rollbacks - A rollout is when a new version of a application is deployed. This triggers a deployment method in K8 called a rollout that deploys new containers with the updated version of the application container. - Introducing the `rollout` sub-command, to review the status of a rollout. ``` kubectl rollout status deployment/$APP ``` - To view the revisions deployed. ``` kubectl rollout history deployment/$APP ``` ## Deployment Strategy - Recreate strategy. - Destroy the existing PODs & deploy new ones with updated application. - Drawback is application downtime. - Rolling update strategy. - Destroy a POD, one at a time & deploy a new one with updated application. - This is the default strategy for deployments. - Update the deployment by modifying the deployment file, (changing whatever option/values.). - In this example, we'll update the version of the image. - Use the `kubectl apply` command to update the deployment. ``` kubectl apply -f $DEPLOYMENT ``` - The `kubectl set` command can also be used to update the image version. - Where `nginx-container` is the specific container to update. - Where `nginx:1.9.1` is the version we're specifying. - Note: This does not update the deployment file! Thus if you forget to update the file, it will still use the previous version not the version used in the set command. ``` kubectl set image deployment/$APP nginx-container=nginx:1.9.1 ``` - The strategy used will be shows in the `kubectl describe deployment $DEPOYMENT` command. - Area "StrategyType" - The events logged for the deployment will differ as well. - This is due to how K8 will deploy the containers based on the strategy used. ## Upgrades - A new ReplicaSet is deployed with the updated PODs. - While the new ReplicaSet is being deployed, the ole ReplicaSet PODs are being destroyed. - To verify this, while performing a upgrade, review the ReplicaSets using the command: ``` kubectl get replicasets ``` ## Rollbacks - To revert a update, use the `kubectl rollout undo` sub-commands, ``` kubectl rollout undo deployment/$APP ``` - This will destroy the newly created deployment with the updated version of the app & return it to the previously running version of the application. - Use the `kubectl get replicasets` command to verify the changes have been made. ## Command Summary - Create the deployment. ``` kubectl create -f $DEPLOYMENT.yaml ``` - Get deployments. ``` kubectl get deployments ``` - Update the deployment - Method A - Update deployment file. ``` kubectl apply -f $DEPLOYMENT.yaml ``` - Update the deployment - Method B - Use the `set` sub-command. ``` kubectl set image deployment/$APP nginx-container=nginx:1.9.1 ``` - Get the status of the update. ``` kubectl rollout status deployment/$DEPLOYMENT kubectl rollout history deployment/$DEPLOYMENT ``` - Rollback the deployment upgrade. ``` kubectl rollout undo deployment/$DEPLOYMENT ``` ### Practice Test - Rolling Updates & Rollbacks - Don't forget about the `edit` sub-command. ``` kubectl edit deployment $DEPLOYMENT ``` ## Configure Applications - This is referencing the underlying Docker system for setting what command & arguments to pass to the command. - This is referring to the docker files themselves & the `CMD` directive. - `CMD` directive will get overwritten during manual invocations. - The `ENTRYPOINT` docker command can be used to specify a command to run when: - The `ENTRYPOINT` directive will get arguments appended to the specific command defined in the docker file. - Use both the `ENTRYPOINT` & `CMD` directives to: - Define the command to run with `ENTRYPOINT`directive. - Define the default arguments to pass with `CMD` directive. - e.g. ``` ENTRYPOINT ["sleep"] CMD ["5"] ``` - The `ENTRYPOINT` directive can be overrode with the `--entrypoint` argument when manually running a Docker container. ``` docker run --entrypoint foo $CONTAINER bar ``` ## Commands & Arguments - New term `args` for passing arguments to the underlying Docker container from K8. - This relates to the `CMD` directive in Docker. - The term `command` is for overriding the `ENTRYPOINT` of the Docker container. - These are used in the `container` area of a pod definition file. - Another method for updating the command is to define it as multiple items in the `command` directive: ``` --- apiVersion: v1 kind: Pod metadata: name: ubuntu-sleeper-3 spec: containers: - name: ubuntu image: ubuntu # Take note of this part specifically. How the 'command' is defined. command: - "sleep" - "1200" ``` ## Configure Environment Variables in Applications - Use the `env` property in the pod definition file. - This is a "plain key" value - It is an array, e.g.: ``` env: - name: APP_COLOR value: pink ``` - Additional ways to add environment variables are: - `ConfigMap` - `Secrets` - Example `ConfigMap` ``` env: - name: APP_COLOR valueFrom: configMapKeyRef: name: webapp-config-map key: APP_COLOR ``` - Example `Secrets` ``` env: - name: APP_COLOR valueFrom: secretKeyRef: name: webapp-config-map key: APP_COLOR ``` ## Configuring ConfigMaps in Applications - ConfigMaps are key/value pairs used to share configuration data for PODs - Data is stored in plain text/not encrypted. - The process for management of ConfigMaps is: - Create the ConfigMap - Inject the ConfigMap into the POD - Methods to create: - Imperative: `kubectl create configmap $CONFIG_NAME` - Declarative: `kubectl create -f $FILE_NAME` - Imperative Method using `--from-literal` flag: ``` kubectl create configmap app_config --from-literal=APP_COLOR=red --from-literal=APP_OTHER=disregard ``` - Imperative Method using file: - When using this method, data is stored under the name of the file in the `configmap`. ``` kubectl create configmap app_config --from-file=app-config.properties ``` - Declarative Method, create a `ConfigMap` definition file: ``` apiVersion: v1 kind: ConfigMap metadata: name: app_config data: APP_COLOR: blue APP_MODE: prod ``` - Declarative Method, create a `ConfigMap` from the file: ``` kubectl create -f app-config.yaml ``` - Ensure to use proper naming conventions/schemes when utilizing `ConfigMaps` as this will be a primary method for managing configuration data between applications. - e.g. Consider different configmaps for MySQL, Redis, or NGINX - View configmaps, `cm` is shorthand for `configmaps` : ``` kubectl get cm ``` - Show `ConfigMaps` data, using `describe`: ``` $ kubectl describe cm db-config Name: db-config Namespace: default Labels: <none> Annotations: <none> Data ==== DB_HOST: ---- SQL01.example.com DB_NAME: ---- SQL01 DB_PORT: ---- 3306 BinaryData ==== Events: <none> ``` - Show `ConfigMaps` data, using `get` with `-o wide`: ``` $ kubectl get cm db-config -o yaml apiVersion: v1 data: DB_HOST: SQL01.example.com DB_NAME: SQL01 DB_PORT: "3306" kind: ConfigMap metadata: creationTimestamp: "2024-07-25T05:42:43Z" name: db-config namespace: default resourceVersion: "963" uid: 7effd9c1-8714-43ca-a9b8-4b148184016c ``` ### ConfigMap in PODs - To associate or inject `configmap` data into a POD, add a field called `envFrom` in the `containers` section of the POD definition file. ``` ... truncated for breviety ... spec: containers: - name: foo image: bar ports: - containerPort: 8080 envFrom: - configMapRef: name: app-config ``` - There are 2 methods for injecting or referencing `configmap` data. - Method A: Reference the complete ConfigMap data. ``` envFrom: - configMapRef: name: app-config ``` - Method B: Reference a specific key/value from the ConfigMap data. - This method is somewhat more tedious but offers granular access to data points. ``` env: - name: APP_COLOR valueFrom: configMapKeyRef: name: app-config key: APP_COLOR ``` - Additional method: Volumes, (to be discussed) ``` volumes: - name: app-config-volume configMap: name: app-config ``` ## Configure Secrets in Applications - Secrets are similar to ConfigMaps but the data is stored **encoded**. - There are 2 methods for creating a Secret. - Imperative method using `--from-literal` , (similar to ConfigMap method): ``` kubectl create secret generic app-secret --from-literal=DB_HOST=mysql ``` - Imperative method using `--from-file` method: ``` kubectl create secret generic app-secret --from-file=app_secret.properties ``` - Declarative method, encode the data. - One method is to use the `echo` command, piped to the `base64` command. - `echo -n 'password' | base64` - Declarative method, create the Secret definition file: ``` apiVersion: v1 kind: Secret metadata: name: app-secret data: APP_PASS: $ENC_DATA ``` - Declarative method, create the Secret from the definition file: ``` kubectl create -f secret-data.yaml ``` - Display Secrets: ``` kubectl get secrets ``` - View Secrets without data: ``` kubectl describe secrets app-secrets ``` - View Secrets with data: ``` kubectl get secrets app-secrets -o yaml ``` - To decode `base64` encoded data: ``` echo -n '$ENC_DATA' | base64 --decode ``` ## Secrets in PODs - To associate or inject `secrets` data into a POD, add a field called `envFrom` in the `containers` section of the POD definition file. ``` ... truncated for breviety ... spec: containers: - name: foo image: bar ports: - containerPort: 8080 envFrom: - secretRef: name: app-secrets ``` - There are 2 methods for injecting or referencing `secrets` data. - Method A: Reference the complete Secrets data. ``` envFrom: - secretsRef: name: app-secrets ``` - Method B: Reference a specific key/value from the Secrets data. - This method is somewhat more tedious but offers granular access to data points. ``` env: - name: APP_PASS valueFrom: secretKeyRef: name: app-secrets key: APP_PASS ``` - Additional method: Volumes, (to be discussed) - using volumes with secrets will have it create a file with the values of the content. - Meaning if you have 3 secrets, 3 files will be created. - These files will have the requested info within them. ``` volumes: - name: app-secrets-volume secret: secretName: app-secrets ``` - NOTES: - Encrypt data at rest. Reference docs on how to do it. - https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/ - https://www.youtube.com/watch?v=MTnQW9MxnRI - TODO: https://www.udemy.com/course/certified-kubernetes-administrator-with-practice-tests/learn/lecture/34549248#overview ## Multi Container PODs - Add another object to the `containers` array in the POD definition file. ``` ... truncated for breviety ... spec: containers: - name: foo image: web-app ports: - containerPort: 8080 - name: bar image: log-agent ``` - References: - https://kubernetes.io/docs/tasks/access-application-cluster/communicate-containers-same-pod-shared-volume/ ### Practice Test - The test solution has a shared directory but that wasn't mentioned in the spec itself. - The test still passes without it & just wanted to note for posterity. - I liked this one for showing the ability to deploy simple apps like Kibana & ElasticSearch. ``` --- apiVersion: v1 kind: Pod metadata: name: app namespace: elastic-stack labels: name: app spec: containers: - name: app image: kodekloud/event-simulator volumeMounts: - mountPath: /log name: log-volume - name: sidecar image: kodekloud/filebeat-configured volumeMounts: - mountPath: /var/log/event-simulator/ name: log-volume volumes: - name: log-volume hostPath: # directory location on host path: /var/log/webapp # this field is optional type: DirectoryOrCreate ``` ### Multi-container Pods Design Patterns - This is detailed more in CKAD program. ## Init Containers - An **Init Container** is a special container used to run supporting commands for other containers in the POD. - This includes cloning code or executing other background commands needed for the POD. - The Init Container is similar to defining a normal container except it uses the object `initContainers`. - Like normal containers, multiple Init Containers can be defined. - However they will only run once in sequential order. ``` initContainers: - name: init-myservice image: busybox command: ['sh', '-c', 'git clone <some-repository-that-will-be-used-by-application> ; done;'] ``` - References: - https://kubernetes.io/docs/concepts/workloads/pods/init-containers/ ## Self Healing Applications - Not covered in training due to not being applicable to CKA. - Covered in CKAD # Section 6: Cluster Maintenance ## OS Upgrades - POD eviction timeout. - How long it takes for a POD to be considered inoperable in the cluster.. - default 5 mins to be "dead". - Use the drain feature to evacuate PODs from a host. - this marks the host as not able to be scheduled to. - migrates PODs to a different host. ``` kubectl drain $NODE ``` - To allow a host to be usable for the cluster ``` kubectl uncordon $DONE ``` - To set a host to not be allowed to schedule new PODs use `cordon`: - This does not migrate PODs - This does not shutdown existing PODs ``` kubectl cordon $NODE ``` #### Practice Test - OS Upgrades - For the command `kubectl drain $NODE` make note of the `--force` & `--ignore-daemonsets` options. - However this can be destructive, research more later. - I deleted the PODs but I need to use the options mentioned previously. ## Kubernetes Software Versions - Optional ## Cluster Upgrade Process - Versions of applications can be a minor version off, (1 higher or lower). - This is within the K8 components themselves not 3rd party components. - This is for non-GCP hosted environments. - There are helper commands to assist with upgrades. - `kubeadm upgrade plan` - Update local version of `kubectl` via OS package manager - Then use `kubectl upgrade apply $version` to actually execute upgrade for control plane components. - Then upgrade the `kublet` on each of the nodes using OS package manager on the master node. - Restart kubelet service. - Then upgrade the kublets on the worker nodes. - Drain the node prior to upgrading kubelet. - Upgrade kubeadm & kublet using OS package manager. - Use kubeadm command to upgrade the nodes config to supported version. - Restart kublet service #### Practice Test - Cluster Upgrade Process - This required editing apt sources list... - edit `/etc/apt/sources.list.d/kubernetes.list` - update version, update apt cache & install specific version. - reference upstream K8 docs. - Used apt-cache to get versions ``` apt-cache madison kubeadm kubelet ``` - Allow to be used ``` kubectl uncordon controlplane ``` - Remember to SSH to the node locally - `ssh node01` - perform the OS package upgrade commands - do not use the kubeadm commands as this is a WORKER node not a control plane node. - meaning the kubeadmin upgrade commands are meant only for the control plane nodes & not worker nodes. ## Backup & Restore Methods ### Declarative Objects - Backup declarative files via Github. - Or query the API & store all existing objects. ``` kubectl get all --all-namespaces -o yaml > all-deployed-services.yaml ``` ### ECTD - Since ETCD also contains information about all of the resources/objects in a K8 cluster this can be backed up as well to simplify managing resources. - Reference the data dir option for directory to backup. - e.g. `--data-dir=/var/lib/etcd` - The is a built-in command to take a snapshot of the etcd data set. ``` ETCDCTL_API=3 etcdctl snapshot save snapshot.db ``` - To restore etcd data set using this method. - Stop kube-api service, e.g. `service kube-apiserver stop` - Then restore from the snapshot file. ``` ETCDCTL_API=3 etcdctl snapshot restore snapshot.db --data-dir /path/to/snapshot ``` - For some reason update the systemctl unit file & update the data dir option to use the path to the snapshot backup(?). - then daemon-reload systemd - restart etcd service - restart kube-apiserver service - NOTE: specify the endpoints & tls certs when executing the `etcdctl` commands. - also specify the etcd api version as shown above. #### Practice Test - Backup Methods 1 - Note the location of the TLS certs: - Server: `/etc/kubernetes/pki/etcd/server.crt` - Key: `/etc/kubernetes/pki/etcd/server.key` - CA: `/etc/kubernetes/pki/etcd/ca.crt` - Snapshot the ETCD dataset: ``` ETCDCTL_API=3 etcdctl --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key --cacert=/etc/kubernetes/pki/etcd/ca.crt snapshot save snapshot.db ``` - Restore the ETCD dataset: ``` ETCDCTL_API=3 etcdctl --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key --cacert=/etc/kubernetes/pki/etcd/ca.crt --data-dir /var/lib/etcd-from-backup snapshot restore snapshot.db ``` - Update the ETCD yaml config, `/etc/kubernetes/manifests/etcd.yaml` - Update the volume to use a different directory so we don't overwrite the existing ETCD directory. - The directory will be the one from the restore command, `/var/lib/etcd-from-backup` - Update the `hostPath` as well to match the volume! #### Practice Test - Backup Methods 2 - List clusters command ``` kubectl config get-clusters ``` - Switch cluster command: ``` kubectl config use-context $CLUSTER ``` - We have access to SSH to each cluster from the student node. #### Certificate Exam Tip - Remember that the actual exam will not give feedback if something is correct or not. - For example, use commands such as `kubectl describe pod` to ensure you used the appropriate name & image. #### References - https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/#backing-up-an-etcd-cluster - https://github.com/etcd-io/website/blob/main/content/en/docs/v3.5/op-guide/recovery.md - https://www.youtube.com/watch?v=qRPNuT080Hk # Section 7: Security ## Security Introduction ## Security Resources - [[Kubernetes+-CKA-+0600+-+Security.pdf]] - [[Attachments/Udemy CKA Resources/Security - Resources/Networking.pdf|Networking]] ## Security Primitives - Authentication, who can access - Files - Usernames & Passwords - Files - Username & Tokens - Certs - External providers - LDAP - Service Accounts - Authorization - RBAC - TLS certs used for communication ## Authentication ### Accounts - Users such as Admins & Developers - Service Accounts for integrations such as bots, etc - Service Accounts can be created in K8S itself. - "Normal" user accounts aren't managed by K8S itself. - These use some form of external authentication method. ### Authentication Methods - Static PW file - Static token file - Certs - Identity Services - LDAP ### Auth Mechanisms - Basic - Create a CSV file in the format, `password,user1,u0001` - Optionally add a 4th column for the group. - e.g. `password1,user1,u0001,group1` - Update the systemctl unit file to include the option, `--basic-auth-file=` - Daemon reload systemd & then restart `kube-apiserver` service. - If using `kubeadm` update the `kube-apiserver` manifest file & include the above mentioned option. - Similar to CSV method, the token method uses a CSV but the file format is slightly different. - `token,user11,u0011,group1` - Option, `--token-auth-file=` - Curl method will use `-H "Authorizaton: Bearer $TOKEN"` #### Note - Not recommended auth mechanism - Use volume mount instead for kubeadm setup - use RBAC ### Example Using Basic Auth - Ignoring for now since this isn't recommended. ## TLS Introduction ### TLS Certs (Pre-Req) - Symmetric encryption, where the same key is used for both the client & server for enc comms. - Since this uses the same key for both client & server, this isn't as secure. - Asymmetric enc, uses a public/private key for each of the client & server. - The keys are *different* & thus allow for secure comms due to the use of separate keys. - Private Key & Public Lock is how this training is describing it. - The Private Key is for your use only. - The Public Lock is for anyone to use with their *own* Private Key. - Example uses SSH. - Notice it creates the public key & private key. - The Public Key is referred to as Public Lock in this instance. - This means while anyone can encrypt data w/ the public key/lock, unless they have the actual Private Key, they'll be unable to de-crypt the data. - Thus in this context, the Public Key/Lock is stored on the server & the Private Key is will be used to "unlock" it per se. - I actually like how they break this down despite it not using the technically correct phrases/words. - Example OpenSSL commands - Generate private key: - `openssl genrsa -out my-bank.key 1024` - Pull out Public Key from it: - `openssl rsa -in my-bank.key -pubout > mybank.pem` - Generate a CSR: - `openssl req -new -key my-bank.key -out my-bank.csr -subj "/C=US/ST=WA/O=MyOrg, Inc./CN=mydomain.com"` - send CSR to CA - They give an example of a MITM, Man-In-The-Middle Attack. - They explain basic TLS/SSL cert usage or PKI. ## TLS in Kubernetes - All components use TLS certs for comms. - Going thru each of the server components TLS usage. - kube-apiserver, apiserver.key & crt - etcd, etcd.key & crt - kublet, kublet.key & crt - Client component TLS usage. - Users/Admins, "" - kube-scheduler, "" - kube-controller-manager, "" - kube-proxy, "" ## TLS in Kubernetes - Cert Creation - We're going thru the whole process with creation of Self-Signed CA, server & client certs. ### Create Self-Signed CA Cert - Generate CA key cert. ``` openssl genrsa -out ca.key 2048 ``` - Create CSR. ``` openssl req -new -key ca.key -subj "/CN=KUBERNETES-CA" -out ca.csr ``` - Sign the CSR & return self-signed CA cert. ``` openssl x509 -req -in ca.csr -signkey ca.key -out ca.crt ``` ### Create Self-Signed Admin cert - Generate Admin user key cert. ``` openssl genrsa -out admin.key 2048 ``` - Create Admin user CSR & specify the group details in CN. ``` openssl req -new -key admin.key -subj "/CN=kube-admin/O=system:masters" -out admin.csr ``` - Sign the CSR & return the self-signed Admin user cert. ``` openssl x509 -req -in admin.csr -CA ca.crt -CAkey ca.key -out admin.crt ``` ### Create Self-Signed Component cert - For these you'll follow same steps except update the cert subject to use the component name. - `/O=system:kube-scheduler` - `/O=system:kube-controller-manager` - `/O=system:kube-proxy` ### Cert Usage - These certs can be used by manually passing to curl or using a `kube-config.yaml` file. - Ensure to copy the CA cert to the nodes/hosts. ### Create Self-Signed Client certs #### ETCD - Follow same steps as used for Admin cert. - Generate a peer cert if using HA config. - Update the etcd config options to reference the certs. - `--key-file=` - `--cert-file=` - `--trusted-ca-file=` - `--peer-cert-file=` - `--peer-client-cert-auth=true` - `--peer-key-file=` - `--peer-trusted-ca-file=` #### kube-api server - We will set aliases for the api-server. - This requires creating a `openssl.cnf` file. ``` [req] req_extensions = v3_req distinguished_name = req_distinguished_name [ v3_req ] basicConstraints = CA:FALSE keyUsage = nonRepudiation subjectAltNames = @alt_names [ alt_names ] DNS.1 = kubernetes DNS.2 = kubernetes.default DNS.3 = kubernetes.default.svc DNS.4 = kubernetes.default.svc.cluster.local IP.1 = 10.x.x.1 IP.2 = 192.x.x.1 ``` - Then create the CSR. ``` openssl req -new -key apiserver.key -subj "/CN=kube-apiserver" -out kube-apiserver.csr -config openssl.cnf ``` - Sign the key using the config. ``` openssl x509 -req -in apiserver.csr -CA ca.crt -CAkey ca.key -out apiserver.crt ``` - Update the systemd kube-apiserver unit file & reference the created cert files. ``` --client-ca-file=/var/lib/kubernetes/ca.pem --tls-cert-file=/var/lib/kubernetes/apiserver.crt --tls-private-key=/var/lib/kubernetes/apiserver.key ``` - Also include TLS certs for ETCD as well. ``` --etcd-cafile=/var/lib/kubernetes/ca.pem --etcd-certfile=/var/lib/kubernetes/apiserver-etcd-client.crt --etcd-cafile=/var/lib/kubernetes/apiserver-etcd-client.key ``` - Also include the kublet TLS certs. ``` --kublet-certificate-authority=/var/lib/kubernetes/ca.pem --kublet-client-certificate=/var/lib/kubernetes/apiserver-kublet-client.crt --kublet-client-key=/var/lib/kubernetes/apiserver-kublet-client.key ``` #### kublet nodes ##### kublet nodes - server cert - These are the clients that run on the nodes to create/manage resources. - TLS certs will use the nodes hostname. - Update `kubelet-config.yaml` config to reference TLS certs. ``` authentication: x509: clientCAFile: "/var/lib/kubernetes/ca.pem" tlsCertFile: "/var/lib/kublet/node01.crt" tlsPrivateKeyFile: "/var/lib/kublet/node01.key" ``` ##### kublet nodes - client cert - kublet CSR, idk about this command/step? - group `system:nodes`? ``` openssl req -new -key kublet-client.key -subj "/CN=kublet/O=system:node:node01" -out kublet-client.csr ``` ## View TLS Certs Details - There are a few methods to deploying/managing TLS certs. - One being the manual method demonstrated previously. - The other using `kubeadm` - For a cluster deployed with `kubeadm` review the POD manifests file to identify the location of the TLS certs actively being used. - e.g. `/etc/kubernetes/manifests/kube-apiserver.yaml` - Review TLS certs using `openssl`. ``` openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text -noout ``` - Verify items such as: - Subject Name - Alternate Names - Cert validity & issuer #### View Logs - Depending on how the components were deployed, to review logs use either: - `journalctl -u etcd-master` - `kubectl logs etcd-master` - If the K8S components are hard down, logs can be review via `docker`. - Obtain all containers ``` docker ps -a ``` - Review logs ``` docker logs $CONTAINER_UUID ``` ### Resources - TLS Spreadsheet - https://github.com/mmumshad/kubernetes-the-hard-way/tree/master/tools ### Practice Test - View Certificate Details - Cert locations kube-apiserver: ``` kubectl describe pods kube-apiserver-controlplane -n kube-system | grep -iE "crt|key" --client-ca-file=/etc/kubernetes/pki/ca.crt --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-account-signing-key-file=/etc/kubernetes/pki/sa.key --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key ``` - Cert locations etcd: ``` kubectl describe pods etcd-controlplane -n kube-system | grep -iE "crt|key" --cert-file=/etc/kubernetes/pki/etcd/server.crt --key-file=/etc/kubernetes/pki/etcd/server.key --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt --peer-key-file=/etc/kubernetes/pki/etcd/peer.key --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt ``` - You can use `crictl` in similar way to `docker` for logging. ``` crictl ps -a crictl logs $CONTAINER_UUID ``` ## Certificates API - Management of the certs can be done via this API. 1. Create `CertificateSigningRequests` objects 2. Review Requests 3. Approve Requests 4. Share User TLS certs - Overall TLS cert creation process using the API: 1. `openssl genrsa -out patrick.key 2048` 2. `openssl req -new -key patrick.key -subj "/CN=patrick" -out patrick.csr` 3. Then this will use the Cert API using a CSR manifest. 1. See **CSR Obeject example** below. 4. base64 encode the file & use that for the `request` object in the manifest file. 1. `cat patrick.csr | base64` 5. View CSR command: `kubectl get csr` 6. Approve the CSR command: `kubectl certificate approve patrick` 7. Obtain the cert using the get CSR command: `kubectl get csr patrick -o yaml` 1. Decode the base64 encoded output & give to user. 2. `echo $BASE64_ENCODED_DATA | base64 --decode` - The `kube-controller-manager` component manages certs. - **CSR Object example** ``` apiVersion: certificates.kubernetes.io/v1 kind: CertificateSigningRequest metadata: name: patrick spec: expirationSeconds: 600 usages: - digital signature - key encipherment - server auth request: $BASE64_ENCODED_CSR ``` ## KubeConfig - A **KubeConfig** file allows you to set information such as TLS cert & key files to use as well as endpoints, etc. - This can be passed to `kubectl` via the `--kubeconfig $config` option. - The file will contain the command line options normally that would be passed directly to `kubectl` but in YAML format. - See **KubeConfig example** - By default it references `$HOME/.kube/confg` - Three areas are: - clusters, the available clusters you have access to. - contexts, is what associates the **clusters** to the **users**. - users, the users to access the clusters. these users will have varying access depending upon the cluster specific to the user. - This isn't *configuring* anything and merely is referencing existing user/cluster info. - View current config command: - `kubectl config view` - `kubectl config view --kubeconfig=$config` - view a specific KubeConfig file. - Switch context config command: - `kubectl config user-context $USER@CLUSTER_CONTEXT` - Namespaces can be defined as well in the **context** area of the KubeConfig file. - NOTE: For TLS certs, specify the full path to the file as a best practice. - The `certificate-authority-data` field can use base64 encoded cert data to replace the `certificate-authority` field. **KubeConfig example** ``` apiVersion: v1 kind: Config current-context: my-k8-playground clusters: - name: my-k8-playground cluster: certificate-authority: ca.crt server: $URL contexts: - name: my-k8-admin@my-k8-playground context: cluster: my-k8-playground user: my-k8-admin namespace: monkeybars users: - name: my-k8-admin user: client-certificate: admin.crt client-key: admin.key ``` ### Practice Test - KubeConfig - Keep it simple & use what I learn. - For instance, instead of editing the file, use the `--kubeconfig` option instead. - Unless directed otherwise :) ## Persistent Key/Value Store - To be demonstrated later :sadface: - I don't wanna manually deploy ETCD at the moment... ## API Groups - Example API groups are: - `/metrics`, for monitoring health of cluster - `/healthz`, for monitoring health of cluster - `/version`, returns the version of the cluster API - `/logs`, used for 3rd party integration - `/api` -> `v1` core group for base functionality - namespaces, pod, nodes, events, services, configmaps, secrets, etc - `/apis`, the "modern" & updated approach to API management. - components have their own API versions and resources. - components are categorized simpler such as: - `/apps` -> `/v1` -> `/deployments` - `/extensions` - `/storage.k8s.io` - `/authentication.k8s.io` - `/certifcates.k8s.io` - `/networking.k8s.io` - Reference upstream API docs for info. - Query the cluster & it will return the available paths. - `curl -k http:$CLUSTER_IP:6443/` - `curl -k http:$CLUSTER_IP:6443/apis | grep name` - Utilize the `kubectl proxy` command to create an internal proxy that creates a connection allowing to use the credentials stored in the kubeconfig file. - **NOTE**: *Kube Proxy* IS NOT *kubectl proxy* - Kube Proxy is what provides connectivity between nodes & cluster services. - `kubectl proxy` is what provides client authenticated access to an internal HTTP endpoint using the Kube Config credentials. ## Authorization - Remember minimum level of access to the cluster. - Apply logical configurations to user/team specific namespaces. - Authorization mechanisms: - **Node authorization**, this is used for nodes & is applied to a node object when configuring it. This is done by adding it to the `system:node` group. This group grants the **Kublet** access to the resources needed to for a node. - **ABAC**, Attribute Based Authorization Control. This is done via Policy files in json format. This file will have objects for *each user or group* with their supported permissions defined in each policy object. - When changes are made to this policy file, the `kube-apiserver` service will need to be restarted for changes to be applied. - As such, this method can be somewhat difficult to manage. - **RBAC**, Role Based Access Control. With this method you will create a role that has the specific permissions allowed/supported for that role. When users are applied the role, they'll have access as per the role. This method allows for simpler management of permissions as it'll be done at the role level vs each user or group. - **Webhook**. Enables 3rd party integration for authorization/access. - Authorization modes are set on the kube-apiserver via the `--authorization-mode=` option in the systemd unit file. - This supports multiple modes in order of how they are defined in the unit file. ## RBAC - Role Base Access Control - RBAC object example, `developer-role.yaml` ``` apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: developer rules: - apiGroups: [""] resources: ["pods"] verbs: ["list", "get", "create", "update", "delete"] - apiGroups: [""] resources: ["ConfigMap"] verbs: ["create"] ``` - Each rule can be applied to each resource object along with which verbs they are allowed to perform on the resource. - Add the role via `kubectl create -f developer-role.yaml` - Next the role will need to be bound to a user via a role binding file. - e.g. `devuser-developer-binding.yaml` - To specify a different namespace than default, add it to the metadata area in the role binding. ``` apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: devuser-developer-binding subjects: - kind: User name: dev-user apiGroup: rbac.authorization.k8s.io roleRef: kind: Role name: developer apiGroup: rbac.authorization.k8s.io ``` - Create the role binding via `kubectl create -f devuser-developer-binding.yaml` - View roles, `kubectl get roles` - View role bindings, `kubectl get rolebindings` - View role info, `kubectl describe role developer` - View role bindings info, `kubectl describe rolebindings devuser-developer-binding` - To review if you, as a user, has access to a resource, use the command: - Update the verb & resource to view its access. ``` kubectl auth can-i create deployments yes kubectl auth can-i delete nodes no ``` - User impersonation to verify user access, use the `--as` option. - The `--namespace` option can verify the namespace access. ``` kubectl auth can-i create pods --as dev-user kubectl auth can-i create pods --as dev-user --namespace test ``` ### Resource Names - To offer more granular namespace management, you can apply the `resourceNames` object to a role. ``` apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: developer rules: - apiGroups: [""] resources: ["pods"] verbs: ["list", "get", "create", "update", "delete"] - apiGroups: [""] resources: ["ConfigMap"] verbs: ["create"] resourceNames: ["blue", "orange"] ``` ### Practice Test - RBAC - To create a Role: ``` kubectl create role developer --namespace=default --verb=list,create,delete --resource=pods ``` To create a RoleBinding: ``` kubectl create rolebinding dev-user-binding --namespace=default --role=developer --user=dev-user ``` - To view what groups/users are associated with a `role` do a `describe` on the `rolebindings` for that namespace. - W/R/T checking if a user can view `pod` I needed to list the specific POD and not try to list all of them. Remember it works on a least privilege model. ## Cluster Roles & Role Bindings - Cluster Scoped objects are items that are: - available to the cluster as a whole. - resources such as nodes. - usually not able to be managed by a `namespace` itself. - To view the namespaced: ``` kubectl api-resources --namespaced=true ``` - To view non-namespaced objects: ``` kubectl api-resources --namespaced=false ``` - Cluster Roles can be used to manage specific resources. - e.g. Cluster Admin can manage nodes - e.g. Storage Admin can manage Persistent Volumes & Claims - Example Cluster Role definition: ``` apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: cluster-administrator rules: - apiGroups: [""] resources: ["nodes"] verbs: ["list", "get", "create", "delete"] ``` - Example Cluster Role Binding: ``` apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: cluster-admin-role-binding subjects: - kind: User name: cluster-admin apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: cluster-administrator apiGroup: rbac.authorization.k8s.io ``` - The process is similar to user roles & role bindings. - NOTE: In some use-cases you can apply Cluster Scoped resources to a user. When that is done, the user has access to *all* resources within that scope. ### Practice Test - Cluster Roles & Role Bindings - For this one, just be patient and read the questions slowly. I was overthinking it and made a simpler error. - I referenced the user friendly name vs the actual resource name. - For the `get` verb, you can add the `--no-headers` flag to remove the header from output. - Using the `kubectl api-resources` command you can obtain info about the objects. Such as the resource names, versions and shorthand names for certain commands. - I need to learn how to do things using the imperative method to save myself time and reduce mistakes. - Example creating role imperatively: ``` kubectl create clusterrole storage-admin --resources=persistentvolumes,storageclasses --verbs=list,create,get,watch ``` ## Service Accounts - Create service account command: ``` kubectl create serviceaccount $NAME ``` - View service account command: ``` kubectl describe serviceaccount $NAME ``` - View service account secret command: ``` kubectl describe secret $SECRET_NAME ``` - Create token for service account (v1.24+) command: ``` kubectl create token $service-account ``` - Secrets can be exported to PODs as Volume Mounts. - A default service account exists with minimal permissions. - New **containers** object `serviceAccountName`. - A service accounts secrets are automatically mounted when setting this object. - When service accounts area created a token is automagically created for the service account. - This is the older implementation and has been replaced by the `RequestTokenApi`. - The tokens don't (didn't) have an expiration date set. - NOTE: Service Account changes v1.24 - `TokenRequestApi` are the new updated token method. - When defining a object using this method, it uses a **projected mount** that is actually a manner of communicating with the `TokenRequestApi`. - Tokens are no longer automatically created for service accounts. - Use the create token command above for this version. - Defining a secret without a expiration is possible in v1.24 via: ``` metadata: name: $secret_name annotations: kubernetes.io/service-account.name: $NAME ``` ## Image Security - By default they are pulled from Dockers upstream repo. ### Private Repository - You'll need to use authentication to login to the upstream repository. - Credentials for repos are stored & accessed by creating a specific type of secret that will have each of the required items for login. - Create a **docker-registry** secret command: ``` kubectl create secret docker-registry regcred \ --docker-server= \ --docker-username= \ --docker-password= \ --docker-email= ``` - New option **imagePullSecrets** in `spec` section: ``` imagePullSecrets: - name: regcred ``` - The `kublet` will utilize these creds when deploying PODs from that upstream repo with authentication. ### Practice Test - Image Security ## Prerequisite - Security In Docker ### Docker Security - This is a quick re-cap/reminder of basic/core Docker topics with regard to security. - Uses same kernel as host OS. - Not completely isolated as a VM is. - Utilizes namespaces for isolation. - Processes within containers by default are run using root user. - The `--user=$USER` option provides the ability to run as non-root user. - This requires setting `USER` option in Dockerfile. - Then build the image: ``` docker build -t $IMAGE_NAME ./path/to/dockerfile ``` - Using the above method will not require passing `--user=USER` option since it is built into the image itself. - Only specific "Linux Capabilities" are enabled for containers by default. - Additional capabilities can be enabled/disabled via the flag `--cap-add $CAP` & `--cap-drop` - e.g. `MAC_ADMIN` ## Security Contexts - When enabling Linux Capabilities this can be done at the POD or Container level (?). - In later slide it says at the Container level & NOT the POD level. Guess we'll figure out soon enough. - If both are enabled, the Containers setting/configuration overrides the POD setting/configuration. - New **spec** & **containers** object `securityContext`. - Similar to the `--user` option for Docker itself, this object can be used to define the user to run the process in the container as. ``` ...output truncated for brevity... spec: OR containers: securityContext: runAsUser: 1000 capabilties: add: ["MAC_ADMIN"] ``` ### Practice Test - Security Contexts - Had to delete & recreate the POD to add the `securityContext`. - Had to remove all but the bare minimum from deployment file to get this to work. - The `runAsUser` option will use the `container` defined option over the `spec` defined option in the manifest. - Have to be careful that the damn option goes in the container area as a child object. - Command to run command in container: ``` kubectl exec ubuntu-sleeper -- whoami ``` ## Network Policy - Ingress, traffic incoming to a service/host, e.g. port 80 on a web server. - Egress, traffic outgoing from a service/host, e.g. port 5000 on a API server that the web server communicates with. - This is in reference/perspective of what host is communicating to whatever host/service. - The example provided mentions the following ruleset: - Ingress port 80 from web server - Egress port 5000 to API server from web server - Ingress 5000 from API server - Egress to port 3306 to DB server - Ingress port 3306 from DB server ### Network Security - By default k8s has a "allow all" network policy so that any POD can communicate with any POD or service defined in the cluster. - Meaning in the example above, all the PODs would be able to communicate, e.g. Web to DB, DB to API, etc. - New object definition `NetworkPolicy`. - Example provided mentions defining a policy for the DB server to only allow connections from the API server. - This policy would only be applicable to the POD that is it defined on. - `labels` & `selectors` are used for associating Network Policies to PODs. - Example Network Policy definition: ``` policyTypes: - Ingress ingress: - from: - podSelector: matchLabels: name: api-pod ports: - protocol: TCP port: 3306 ``` - Complete definition file: ``` apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: db-policy spec: podSelector: matchLabels: role: db policyTypes: - Ingress ingress: - from: - podSelector: matchLabels: name: api-pod ports: - protocol: TCP port: 3306 ``` - Create the policy: `kubectl create -f network-policy.yaml` - In this example, this is only applied to **ingress** traffic & not **egress** traffic. - Thus all egress traffic is allowed. - Note: These solutions provide support for Network Policies. - kube-router - calico - romana - weave-net - Note: If a solution does not support Network Policies you will not get an error & it will just create it but not be able to enforce it. ### Developing Network Policies - Example DB Network Policy ``` apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: db-policy spec: podSelector: # the POD to apply policy to. matchLabels: role: db policyTypes: - Ingress ingress: - from: # source of traffic - podSelector: matchLabels: name: api-pod namespaceSelector: matchLabels: name: prod ports: - protocol: TCP port: 3306 ``` - One thing I was initially confused about was when adding network policies, do you need to configure an additional rule for egress that is related to the ingress rule. - No you do not. K8 will automagically allow egress traffic for whatever ingress port. e.g. Allowing ingress to port 3306 will automatically allow egress for it. - However for the DB to access the API, it would need to have a egress rule on the API - New keyword `namespaceSelector`. Used to define the namespace to apply the network policy to. The label used for `matchLabel` must be set in order for this to work appropriately. - This can be used to allow all traffic from a namespace to a resource. - New keyword `ipBlock`. This is used to specify an IP address that is outside of the current cluster. ``` - ingress: - from: - ipBlock: cidr: IP.AD.DR.ESS/32 ``` - Each **element** defined in the `ingress` or `egress` sections are *rules* that comprise the network policies. - These work in a AND/OR method. Thus each rule defined within a policy will be parsed as such. Thus using multiple keywords requires ensuring you understand how they will be parse/processed. - Example egress rule: ``` policyTypes: - Ingress - Egress ingress: - from: - podSelector: matchLabels: name: api-pod ports: - protocol: TCP port: 3306 egress: - to: - ipBlock: cidr: 192.168.1.2/32 ports: - protocol: TCP port: 80 ``` #### Practice Test - Network Policies - Get all network policies on default namespace: ``` kubectl get networkpolicies ``` - Reference upstream docs for "Network Policies" - Question 10 was referencing the whole PODs not just the nodes within. I was confusing this and thus not understanding what was being asked. Also I had done the practice test out of order... ### Kubectx & Kubens - Utilities to simplify K8 namespace/context switching. - https://github.com/ahmetb/kubectx # Section 8: Storage ## Storage Intro - No data ## Intro to Docker Storage - No data ## Storage In Docker - Docker file system directory - `/var/lib/docker` - Layered architecture - For each docker command in the docker file, this relates to the amount of data being stored/written. - When docker is building containers, it'll reference it's cache (if the docker-file is similar), to create the image. Meaning instead of having duplicates of data, it'll instead only create new data based on the differences of the cached image. - This is the "image" layer. Once a image is built, it becomes read-only. - The "container" layer when a container is run, is the layer that is writable. - "COW" - Copy On Write. This is when you edit a file on the container that is from the image itself. e.g. code. Modifying of this data at this layer, does not affect the underlying file in the image itself. - Docker Volumes allow for persistent data storage for files/data that you want to read-write to but is outside of the image & container themselves. - These volumes are created in the directory, `/var/lib/docker/volumes/$VOL` - Creating a docker volume: ``` docker volume create data_volume ``` - Running a container & attaching a volume to it: - The volume is stored in the docker volumes directory on the host as mentioned previously. - This will allow the `/var/lib/mysql` in the container to be persistent & read/writable. - If the volume doesn't exist it will be automatically created. ``` docker run -v data_volume:/var/lib/mysql mysql ``` - To attach a directory from a different location than the default: - This is a bind mount that associates the directory on the host with the specified directory on the container. ``` docker run -v /data/mysql:/var/lib/mysql mysql ``` - Using `-v` is legacy, use `--mount` instead. For example: ``` docker run \ --mount type=bind,source=/data/mysql,target=/var/lib/mysql mysql ``` - The various storage drivers are what support whatever volume management required. ## Volume Driver Plugins in Docker - Storage Drivers are what interact with software such as: - aufs - zfs - btrfs - device mapper - overlay - Volume Drivers are what interact with software such as: - Azure FS - Convoy - Other Block Storage solutions. - The default one used is local and this uses the default docker directory for storing container data. - Volume Drivers are related more to software volumes vs actual file systems. - Attaching a 3rd party volume to a docker instance using the `--volume-driver` option. ``` docker run -it \ --name mysql --volume-driver rexray/ebs --mount src=ebs-vol,target=/var/lib/mysql mysql ``` ## Container Storage Interface (CSI) - This is the standard of how storage interfaces are integrated to K8S. - These are the drivers of whatever storage solution your wanting to use. ## Volumes - Docker containers are meant to be transient upon deployment. - Meaning that the containers & the associated data are removed once the container is deleted. - To have data be persistent, volumes are attached to containers. - PODs in K8S are similar & once a POD is deleted the associated data is as well. ### Volumes & Mounts - Example POD definition with volume: - This method isn't recommended for multi-POD configurations. ``` apiVersion: v1 kind: Pod metadata: name: random-number-generator spec: containers: - image: alpine name: alpine command: ["/bin/sh","-c"] args: ["shuf -i 0-100 -n 1 >> /opt/number.out;"] volumeMounts: # This is what is mounted on the containers. - mountPath: /opt name: data-volume volumes: # This is what actually creates the mount on the host. - name: data-volume hostPath: path: /data type: Directory ``` - Example of using AWS Elastic Storage: ``` volumes: - name: data-volume awsElasticBlockStorage: volumeID: <vol-id> fsType: ext4 ``` ## Persistent Volumes - In the previous lesson, when defining storage this was done at the POD level. - This method doesn't scale in larger environments. - Users would have to manually manage volumes independently & this can increase storage overhead. - Instead, Persistent Volumes offers a simpler way to manage volumes for the cluster as a whole. - A storage pool that users can utilize. - Example PV definition: ``` apiVersion: v1 kind: PersistentVolume metadata: name: pv-vol1 spec: accessModes: - ReadWriteOnce capacity: storage: 1Gi # This part in example is not for a prod deployment. hostPath: path: /tmp/data # OR use this below for prod deployment. awsElasticBlockStore: volumeID: <volume-id> path: /tmp/data fsType: ext4 ``` - Create the PV ``` kubectl create -f pv-definition.yaml ``` - List PV ``` kubectl get persistentvolume ``` ### Persistent Volume Claims - These are what bind the Persistent Volumes (PV) to the PODs themselves. - Every PVC is associated to only ONE PV. - K8S will try to select an appropriate PV to bind to the PVC based on the criteria needed. - However Selectors can be used to target a specific PVC to a PV. - Note a smaller PVC can use a larger PVC if the requested smaller PVC does not exist for usage. - All claims are 1 to 1. - A claim will remain in a *pending* state if no PVs are available. - Example PVC definition: ``` apiVersion: v1 kind: PersistentVolumeClaim metadata: name: myclaim spec: accessModes: - ReadWriteOnce resources: requests: storage: 500Mi ``` - Create the PVC: ``` kubectl create -f pvc-definition.yaml ``` - List the PVC: ``` kubectl get persistenvolumeclaim ``` - To delete a PVC: ``` kubectl delete persistenvolumeclaim myclaim ``` - By default when a PVC is deleted, it is *retained* via the option: - `persistentVolumeReclaimPolicy: Retain` - Other options include: *Delete* to return the storage or *Recycle* to have the data scrubbed before being returned for use into the cluster. ### Using PVCs in PODs - https://kubernetes.io/docs/concepts/storage/persistent-volumes/#claims-as-volumes - Example PVC POD definition: ``` apiVersion: v1 kind: Pod metadata: name: mypod spec: containers: - name: myfrontend image: nginx volumeMounts: - mountPath: "/var/www/html" name: mypd volumes: - name: mypd persistentVolumeClaim: claimName: myclaim ``` ### Practice Test - PV & PVC - Running a command within the POD: ``` kubectl exec webapp -- cat /log/app.log ``` - Recreate a POD: ``` kubectl replace -f webapp.yaml --force ``` - Remember the difference between: - `volumeMounts` that are for the PODs. - `volumes` that are on the host that are exported to the PODs. - Note that *accessModes* being mismatched will cause a PVC & PV to not bind. They must be the same! - When a PV is set to *retain* the volume remains but is not usable. (?) - Remember you can use the official upstream documentation. ## Application Configuration - Reference the [[#Volumes]] section as this explains how to use volumes in POD definition files for application deployments. ## Additional Topics - Additional topics such as StatefulSets are out of scope for the exam. However, if you wish to learn them, they are covered in the Certified Kubernetes Application Developer (CKAD) course. ## Storage Classes - Overall the process for volume management is: 1. Create the PV as described in [[#Persistent Volumes]] 2. Create the PVC as described in [[#Persistent Volume Claims]] 3. Attach the PVC as described in [[#Using PVCs in PODs]] ### Static Provisioning - Static Provisioning is when using a 3rd party storage provider such as Google Cloud, you'd need to have created the volume beforehand & then create the PV to use it. - You will need to ensure you reference the name of the object in the storage provider in the definition file. - e.g. If you named the GCloud object `pd-disk` then in the definition file you would specify that same name. ``` ... output truncated for brevity ... gcePersistentDisk: pdName: pd-disk fsType: ext4 ``` ### Dynamic Provisioning - Dynamic Provisioning is the ability for K8S to automatically create the volumes for you to the underlying 3rd party storage provider. - This is done using a **Storage Class Object**. - Example Storage Class Object: ``` apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: google-storage provisioner: kubernetes.io/gce-pd parameters: type: pd-standard ``` - Using this method the process for volume management would be: 1. Create the SC as described in [[#Dynamic Provisioning]] 2. Create the PVC as described in [[#Persistent Volume Claims]] & add the new option `storageClassName: google-storage` to the *spec* area in the definition file. 3. Attached the PVC as described in [[#Using PVCs in PODs]] - Example PVC using a SC: ``` apiVersion: v1 kind: PersistentVolumeClaim metadata: name: myclaim spec: accessModes: - ReadWriteOnce storageClassName: google-storage resources: requests: storage: 500Mi ``` - Note the PV is still created but it is done for you via the SC. - A feature using this method is that you can create specific types of SCs depending on use-case. - For example, a SC for slow, fast or high-availability. ### Practice Test - Storage Class - Make sure to reference the correct `storageClassName` when defining it in the definition file. - When the *WaitForConsumer* option is defined for a PV, the PVC will not be created until a POD is deployed to utilize it. # Section 9: Networking ### Prerequisite - Switching Routing - Very introductory. - Some basic Linux networking commands: ``` ip link # show network interfaces ip addr add $IP dev eth0 ip route add $SUBNET via $GATEWAY_IP cat /proc/sys/net/ipv4/ip_forward ``` ### Prerequisite - DNS - Very introductory. ### Prerequisite - CoreDNS - https://github.com/kubernetes/dns/blob/master/docs/specification.md - https://coredns.io/plugins/kubernetes/ ### Prerequisite - Network Namespaces - The following describes 1st how to manage/modify Network Namespaces & then how to connect them to each other directly. - The 2nd part describes how to create a Linux Bridge Network for the Network Namespaces to connect to in order to connect to each other. - This is similar to how a physical network with switching would operate. ### Single Network Namespace - Create Network Namespace ``` ip netns add $NAMESPACE_NAME ``` - List NS ``` ip netns ``` - Exec in Network Namespace ``` ip netns exec $NAMESPACE_NAME ip link ip -n $NAMESPACE_NAME link ``` - Create Network Pipe or Connection between Namespaces ``` ip link add veth-red type veth peer name veth-blue ``` - Attach the Network Pipe to Namespace ``` ip link set veth-red netns red ip link set veth-blue netns blue ``` - Add IP to Namespace ``` ip -n red addr add $IP dev veth-red ip -n blue addr add $IP dev veth-blue ``` - Set Namespace link up ``` ip -n red link set veth-red up ip -n blue link set veth-blue up ``` - Ping between Namespaces ``` ip netns exec red ping $BLUE_IP ip netns exec blue ping $RED_IP ``` ### Bridged Network Namespace - Create Linux Bridge Network ``` ip link add v-net-0 type bridge ``` - Set Linux Bridge Interface UP ``` ip link set dev v-net-0 up ``` - Delete Network Pipe for Namespace - Blue delete not needed since the delete automatically removes the blue pipe. ``` ip -n red link del veth-red ``` - Attach Bridge to Namespace ``` ip link add veth-red type veth peer name veth-red-br ip link add veth-blue type veth peer name veth-blue-br ``` - Attach Network Pipe to Namespace & to Bridge ``` ip link set veth-red netns red ip link set veth-red-br master v-net-0 ip link set veth-blue netns blue ip link set veth-blue-br master v-net-0 ``` - Add IP to the Network Namespaces connected to the Bridge & set up link ``` ip -n red addr add $IP dev veth-red ip -n red link set up veth-red ip -n blue addr add $IP dev veth-blue ip -n blue link set up veth-blue ``` - Add IP to the Bridge interface itself for connectivity between the host & the Network Namespaces. ``` ip addr add $IP dev v-net-0 ``` - Add route from Namespaces to the Bridge interface for external connectivity. ``` ip netns exec blue ip route add $SUBNET via $BRIDGE_IP ``` - Enable routing connectivity via NAT using IPTABLES. - This allows external connectivity & external hosts will receive the traffic as coming from the host. ``` iptables -t nat -A POSTROUTING -s $SUBNET -j MASQUERADE ``` - Set the host as the gateway for external connectivity to the Internet ``` ip netns exec blue ip route add default via $HOST_IP ``` - Port forwarding to the Namespaces using IPTABLES ``` iptables -t nat -A POSTROUTING --dport 80 --to-destination $NS_IP:80 -j DNAT ``` ## Network FAQ - Remember to ensure to set the correct netmask for the subnet for the Namespace. - Verify that IPTABLES has rules for NAT. ## Prerequisite - Docker Networking - Run docker container without networking ``` docker run --network none nginx ``` - Run docker container with host based networking. - This will operate as the port on the container will appear to be from the host itself. - Trying to run another container using the same port will fail as the port will already in use by the other container. ``` docker run --network host nginx ``` - List docker networks ``` docker network ls ``` - The `docker0` interface is a bridge. This is similar to previous lecture where we added bridges manually. - When listing these Namespaces on the host, they will be have UUID type names defined. - Docker will automatically create Namespace interfaces for the containers. - These will be on the 172.x.x.x/16 subnet. - Docker can map ports within the containers to ports on the host. - Port 8080 is on the host & will forward connections to port 80 on the container. ``` docker run -p 8080:80 nginx ``` - The port mapping is essentially done via iptables. ``` iptables -t nat -A DOCKER -j DNAT --dport 8080 --to-destination $CONTAINER_IP:80 ``` - List the iptables rules for docker: ``` iptables -nvL -t nat ``` ## Prerequisite - CNI - Container Network Interface is a set of standards that defines how Container Networking should be implemented. - This allows for various solutions to implement container network management using these standards. - For instance how to add, remove, or modify container networking is defined by the CNI. - These programs or scripts are "plugins" to K8S for container network management. - Any runtime should work with any plugin. - Docker utilizes CNM or the Container Network Model. - Docker does not use the CNI. - Thus docker doesn't support CNI plugins for container network management. - The work around is to create a docker container without networking, e.g. `--network=none` and then manually invoke the CNI plugin to then manage the container network. - K8S essentially does this as well when using Docker. - It will create the container without Docker networking & the call the currently enabled plugin configured for the cluster. ## Cluster Networking ### Networking Cluster Nodes - A deployment of any sort for a cluster node should have at least one interface, unique MAC address, IP address & FQDN. - `kube-api` access on the master listens on port 6443. - `kube-scheduler` listens on port 10259 - `kube-controller-manager` listens on port 10257 - `kublets` on the master & nodes listen on port 10250. - worker nodes expose user services on ports 30000-32767 - `etcd` service listens on port 2379 - multiple master nodes require the mentioned ports to be open as well as port `2380` for the `etcd` peer-to-peer to communicate. - ref: upstream K8S docs, "check-required-ports" ### Network Commands - `ip link` - show interfaces - `ip addr` - show IP addresses - `ip addr add $IP/$SUBNET dev $dev` - add IP address to interface - `ip route` - show IP routing - `route` - legacy IP routing command - `ip route add $NETWORK/$SUBNET via $GATEWAY_IP` - add route for subnet. - `cat /proc/sys/net/ipv4/ip_forward` - Check if forwarding is enabled, `1` means it is. - `arp` - Show arp tables, e.g. what interface is responding to what IP address. - `netstat -plnt` - show running network services ## Notes about CNI & CKA Exam - The exam allows you to use any of the supported 3rd party plugins BUT the upstream docs DO NOT provide direct information on how to implement/use them. - NOTE: In the official exam, all essential CNI deployment details will be provided. ## Practice Test - Explore Environment - Used `kubectl describe nodes controlplane` to get `InternalIP` of control plane node. - To get `node01` MAC addr, use above command to describe `node01` & then SSH to it. - Then use standard `ip l l` command to get MAC addr. - Question mentioned `containerd` interface and I guessed it to be `cni0` - Question about `etcd` & which port had the most connections, used `netstat -pnt | grep etcd` to identify which port had most connections. ## POD Networking - POD Networking Model - Every POD should have an IP. - Every POD should be able to communicate with every other POD in the same node. - Every POD should be able to communicate every other POD on other nodes without NAT - POD Network Example - Each Node has its own IP in 192.168.1.0/24 subnet. - We will create a Network Namespace on each node for the PODs - A Bridged network namespace called `v-net-0` - `ip link add v-net-0 type bridge` - Bring the Bridge interfaces online/up - `ip link set v-net-0 up` - Assign an IP to each of the Bridges, these will use `10.244.0.1-3/24` respectively. - `ip addr add 10.244.0.1/24 dev v-net-0` - A script called `net-script.sh` will be created & used to perform the manual adding of addresses to containers. - Reference [[#Single Network Namespace]] & [[#Bridged Network Namespace]] ``` # create veth pair ip link add veth-$con type veth peer name $NETNS # attach veth pair ip link set veth-$con netns $NETNS ip link set veth-$con master $NETNS # assign IP ip -n $NETNS addr add $IP dev veth-$con ip -n $NETNS route add $SUBNET via $BRIDGE_IP # set interface up ip -n $NETNS link set veth-$con up ``` - To enable connectivity between the PODs, on NODE1 add a route to the network on NODE2, `192.168.1.12` - NODE1: `ip r add 10.244.2.2 via 192.168.1.12` - We will do similar on all the NODEs to enable routing between PODs. - Otherwise utilize a router to do this on our behalf. - CNI can be used to help utilize the script to perform this on our behalf. ## CNI In Kubernetes ### Configuring CNI - CNI is configured in the Container Runtime, e.g. containerd, cri-o - These will reference the specified CNI plugin located in `/opt/cni/bin` - How they are configured to be used is located in `/etc/cni/net.d` - The plugin used will be referenced by alphabetical order, e.g. `10-bridge` - This configuration file will specify the data needed to create/manage POD networking on the host/cluster. ## Note CNI Weave - Utilize the following to install Weave. ``` kubectl apply -f https://github.com/weaveworks/weave/releases/download/v2.8.1/weave-daemonset-k8s.yaml ``` - Weave upstream references: - Reference links: - https://www.weave.works/docs/net/latest/kubernetes/kube-addon/#-installation - https://github.com/weaveworks/weave/releases ## CNI Weave - A POD can be attached to multiple bridged networks. - The route it takes depends on which of the bridges is configured to route for that network. e.g. docker or weave ### Deploy Weave - Weave Peers are PODs running in the `kube-system` POD. - `kubectl get pods -n kube-system` - To view weave logs: - `kubectl logs weave-net-$UUID weave -n kube-system` ## Practice Test - Deploy Network Solution - The Weave Manifest file was already created & we merely needed to execute: - `kubectl create -f weave/weave-daemonset-k8s.yaml` - Use `apply` instead per the solution. - Reference upstream K8S docs and find Weave Net area. - select it and then install it as per documentation mentions to do. - Command to provide info on ConfigMaps: - `kubectl describe cm kube-proxy -n kube-system` ## IP Address Management - Weave ### IPAM CNI - Nodes are excluded from this portion. - Weaveworks by default uses subnet `10.32.0.0/12` - The weave net peers will split up addresses among themselves to provide to containers. ## Practice Test - Networking Weave - Checked subnet being used via `kubectl get cm kube-proxy -n kube-system`. - then referenced `clusterCIDR` - You can also check the **weave** PODs logs for info: - `kubectl logs weave-net-jmtkg -n kube-system | grep ipalloc-range` - To create a test POD manifest that uses busybox image: ``` kubectl run busybox --image=busybox --dry-run=client -o yaml -- sleep 1000 > busybox.yaml ``` - Then to deploy to a specific node edit the manifest & add `nodeName: $NODE` - NOTE: Remember you can run commands in a POD via: ``` kubectl exec busybox -- $COMMAND ``` ## Service Networking - Unlike POD networking where instances connect directly to each other, Service networking is how you expose a application/service to other PODs and/or Services. - Services will have an additional IP address associated with them that will forward traffic to the underlying POD. - Services are accessible from all PODs regardless of which nodes they are on. - Consider that while a POD is hosted on a **node**, that a Service is hosted on the **cluster**. - The Service is accessible only from the cluster itself using a **ClusterIP**. - For a Service to be accessible outside of the cluster a **Node Port** is used that provides an IP **and** port of the application/service for access. - Consider it as such: - **ClusterIP** is for inter-cluster communication. - **NodePort** is used for out of cluster communication. - Services unlike POD Network Namespaces do not have an actual process that is actually running the Service. It is a virtual object. - This virtual object is essentially a VIP that gets traffic forwarded from it to the underlying PODs that actually provide the Service. - Services are assigned an IP **and** the port associated with the POD providing the service, e.g. port `80` or `443`, etc. - The kube-proxy service is what configures this by using either a `userspace`, `ipvs` or `iptables` method. - By default it uses `iptables`. - K8S commands, get Services. - `kubectl get service` - This is configured for the `kube-apiserver` via, **--service-cluster-ip-range** option. - By default it uses `10.0.0.0/24` - `ps faux | grep -E "\-\-service\-cluster\-ip\-range"` - Network overlap shouldn't occur when setting/defining this option. - The Service iptables rules can be viewed via `iptables -L -t nat | grep $SERVICE` - `kubectl logs kube-proxy -n kube-system` - kube-proxy logging can yield this info as well, default directory `/var/log/kube-proxy.log` - To see which proxy method is being used: - `kubectl logs kube-proxy -n kube-system | grep Using` ## Practice Test - Service Networking - 1st question was odd. - Had to do `ip a | grep eth0` - Then use `ipcalc -b $IP` - Solution used `kubectl get nodes -o wide` then reference **Internal IP** values. - Since the network is configured with `weave`, reference [[#Practice Test - Networking Weave]] - Reference the manifest in `/etc/kubernetes/manifests/kube-apiserver.yaml` - Then view the value for option `--service-cluster-ip-range` - To see how the `kube-proxy` PODs are deployed: - `kubectl describe pod kube-proxy-fj7mh -n kube-system | grep "Controlled By:"` - View output from `kubectl get all --all-namespaces`. - Then view the POD name, e.g. `daemonset.apps/kube-proxy` ## DNS In Kubernetes ### Cluster DNS - DNS in K8S is configured per namespace. Meaning that for instance, in the default namespace you would just reference the service, e.g. "web-service" or IP directly. - Service namespace resolution. - However for other namespaces you would call the service plus the namespace, e.g. "web-service.apps", where "apps" is the namespace or sub-domain that the "web-service" resides in. - All PODs are grouped together by a sub-domain to reference the associated namespace. - For services this will include the additional sub-domain of `svc`. So for the `web-service` it would be `web-service.apps.svc` to access it. - Additionally the root domain of `cluster.local` is used by default & thus you can reference the service as `web-service.apps.svc.cluster.local` as the FQDN. - POD DNS resolution is not enabled by default. - For PODs a standard resource record is not created but one that uses the IP. For example, if the IP is `10.0.0.1` the associated DNS resource record will be `10-0-0-1`. ## CoreDNS in Kubernetes ### How Kubernetes Implements DNS? - For PODs a standard resource record is not created but one that uses the IP. For example, if the IP is `10.0.0.1` the associated DNS resource record will be `10-0-0-1`. - CoreDNS is the default name service used internally for K8S. - It's deployed as a replicaset in a deployment. - The config is located in `/etc/coredns/Corefile` - This is passed into K8S as a `ConfigMap` object. - `kubectl get cm -n kube-system` - To utilize this NS for PODs a K8S Service is created. - `kubectl get service -n kube-system` - The `kubelets` are responsible for defining this within the PODs themselves. - reference: `/var/lib/kubelet/config.yaml` - PODs RRs aren't accessible via `host` using shorthand as those records aren't configured within the search domain. - Thus you will have to call the complete POD hostname, e.g. `10-0-0-1.default.pod.cluster.local` ## Practice Test - Explore DNS - To identify the `Corefile` used for CoreDNS, reference the args passed to the CoreDNS deployment. - `kubectl describe deployments.apps -n kube-system` - To run command from a POD: ``` kubectl exec $POD -- $COMMAND ``` - For the question related to the `webapp` service accessing the `mysql.payrool.svc` I had to edit the POD itself and not the service. - Well seems that I should have edited the `deployment` instead ``` kubectl edit deploy webapp ``` ## Ingress - This is to demonstrate the differences between Services & Ingress. When to use which based upon the use-case or need. - In the example provided, it's a simple webapp & mysql db. - You create a Service for MySQL for the webapp to connect. - You create a Deployment with a `NodePort` for external access. - However this results in users having to access the Service as `http://my-site.com:38080` which isn't desirable. - If using a provider such as GCP you will instantiate a Service with a GCP LB. The Deployment will then instantiate a LB using GCP so that users can access the Service as `http://my-site.com` without needing to use the port. - The example is expanded upon by adding additional Services, e.g. a video service. - This can become complexicated when attempting to use multiple LBs for each service. - Thus concept of Ingress is used to provide URL based load-balancing for the Services. - Ingress can be considered a L7 load-balancer to manage traffic based upon the URL referenced. - With Ingress you will still need to expose the Service as a `NodePort` or a GCP LB, (or whatever service provider being used). - Ingress utilizes solutions such as NGINX, HAproxy or traefik for actual traffic management. This is called a **Ingress Controller**. - The solution will be deployed and then you configure it as per that solutions best practices & needs. - The configuration for this is called **Ingress Resources**. - By default there is no Ingress Controller enabled using the mentioned solutions. - GCP & NGINX are the upstream supported solutions for an Ingress Controller. - The controllers are deployed as another Deployment. - The NGINX controller is a specially created instance specifically for K8S. - A ConfigMap object is used to configure this. - Reference [[Ingress.pdf]] for example on configuring CM object for this. ``` apiVersion: extensions/v1beta1 kind: Deployment metadata: name: nginx-ingress-controller spec: replicas: 1 selector: matchLabels: name: nginx-ingress template: metadata: labels: name: nginx-ingress spec: containers: - name: nginx-ingress-controller image:quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.21.0 args: - /nginx-ingress-controller - --configmap=$(POD_NAMESPACE)/nginx-configuration env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace ports: - name: http containerPort: 80 - name: https containerPort: 443 ``` - Then we create a `NodePort` service to expose this for access. - fix formatting... ``` apiVersion: v1 kind: Service metadata: name: nginx-ingress spec: type: NodePort ports: - port: 80 targetPort: 80 protocol: TCP name: http - port: 443 targetPort: 443 protocol: TCP name: https selector: name: nginx-ingress ``` - Then we create a `ServiceAccount` to associate with this Service. - This account will have Roles, ClusterRoles, & RoleBindings configured in order to appropriately manage the related resources within its Cluster scope. ``` apiVersion: v1 kind: ServiceAccount metadata: name: nginx-ingress-serviceaccount ``` - Thus in total we will have the following object: - Deployment - Service - ConfigMap - Auth (ServiceAccount) - Ingress Resource is what is configured to define the routing of requests depending on URL or Domain Name requested. - This is configured as any other object using a definition file. - This will define the Service & Port to route requests to. - To list Ingress objects: ``` kubectl get ingress ``` - Ingress Resource Rules are used to define specific routing for the backend applications/services. - e.g. each domain name used by the service will have a rule. - Within each you can can configure additional "paths" for routing based on URL. - To describe Ingress Resources: ``` kubectl describe ingress $INGRESS_RESOURCE ``` - Without defining a `host` option in the Ingress Resource this will be considered a wild-card & match all requests & route them appropriately. ## Article: Ingress - These are updates to the K8S training with regards to defining Ingress objects. - Imperatively creating an Ingress object: ``` kubectl create ingress $INGRESS_NAME --rule="host/path=service:port" ``` - Example of an Ingress object: ``` kubectl create ingress ingress-test --rule="wear.my-online-store.com/wear*=wear-service:80" ``` - Upstream documentation: - https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#-em-ingress-em- - https://kubernetes.io/docs/concepts/services-networking/ingress - https://kubernetes.io/docs/concepts/services-networking/ingress/#path-types ## Ingress - Annotations & Rewrite-Target - Upstream documentation: - https://kubernetes.github.io/ingress-nginx/examples/ - https://kubernetes.github.io/ingress-nginx/examples/rewrite/ - This is a quick breakdown of how the Rewrite functionality works with the NGINX Ingress plugin. ## Practice Test - Ingress - 1 - Namespaces: - app-space - ingress-nginx - Retrieve all objects in the app-space namespace: - `kubectl get all -n app-space` - PODs: ``` kubectl get pods -n app-space NAME READY STATUS RESTARTS AGE default-backend-78f6fb8b4-q2vvt 1/1 Running 0 3m56s webapp-video-74bdc86cb8-dmbp6 1/1 Running 0 3m56s webapp-wear-6f8947f6cc-g45vw 1/1 Running 0 3m56s ``` - Replicasets: ``` kubectl get replicasets -n app-space NAME DESIRED CURRENT READY AGE default-backend-78f6fb8b4 1 1 1 4m webapp-video-74bdc86cb8 1 1 1 4m webapp-wear-6f8947f6cc 1 1 1 4m ``` - Deployments: ``` kubectl get deployments -n app-space NAME READY UP-TO-DATE AVAILABLE AGE default-backend 1/1 1 1 4m5s webapp-video 1/1 1 1 4m5s webapp-wear 1/1 1 1 4m5s ``` - Services: ``` kubectl get services -n app-space NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default-backend-service ClusterIP 10.100.188.224 <none> 80/TCP 6m27s video-service ClusterIP 10.100.245.53 <none> 8080/TCP 6m27s wear-service ClusterIP 10.110.34.180 <none> 8080/TCP 6m27s ``` - Ingress: ``` kubectl get ingress -n app-space NAME CLASS HOSTS ADDRESS PORTS AGE ingress-wear-watch <none> * 10.100.227.180 80 5m35s ``` - Question 22, add the pay-service to the Ingress object `ingress-wear-watch` - Service: pay-service - Deployment: webapp-pay - Port 8080 - Question 22 Solution: - You can't create a Ingress for another namespace in other namespaces. - This is a best practice as well in that you want to isolate which namespaces are being used for whatever objects. - In this instance, `app-space` namespace shouldn't directly reference resources in the `critial-space` namespace. - Don't forget you can use the help option to get information on using the objects: - `kubectl create ingress -h` - To create the `pay-service` Ingress object: - You will also add an `annotation` option to the object in order to have it rewrite requests to `/pay` to go to the apps root (`/`) - Annotations go in the `metadata` section of the definition. ``` kubectl create ingress ingress-pay -n critical-space --rule="/pay=pay-service:8282" ``` - Adding Rewrite annotation to the object: ``` ...truncated for brevity... metadata: annotations: nginx.ingress.kubernetes.io/rewrite-target: / ``` ## Practice Test - Ingress - 2 - All namespaces: ``` kubectl get namespaces NAME STATUS AGE app-space Active 83s default Active 4m34s kube-flannel Active 4m31s kube-node-lease Active 4m34s kube-public Active 4m34s kube-system Active 4m34s ``` - Create a Namespace object: ``` kubectl create namespace ingress-nginx ``` - Create a ConfigMap object: ``` kubectl create cm ingress-nginx-controller -n ingress-nginx ``` - Create ServiceAccount objects: ``` kubectl create serviceaccount ingress-nginx -n ingress-ngix kubectl create serviceaccount ingress-nginx-admission -n ingress-ngix ``` - Create separate namespaces for ingress obejects. - Question 6 ensure: - object names are properly set - objects referenced are properly capitalized/defined - Question 7 - Create ingress for backend services: - you can use multiple rules when creating ``` kubectl create ingress wear-service --rule="/wear=wear-service:8080" --annotation "nginx.ingress.kubernetes.io/rewrite-target: /" -n app-space kubectl create ingress video-service --rule="/watch=video-service:8080" --annotation "nginx.ingress.kubernetes.io/rewrite-target: /" -n app-space ``` - This is slightly different than the actual questions asked but is still useful. - Question 7 - They used the `expose` command on the deployment ingress-controller object of instead & then edited it to add the nodePort. ``` kubectl expose deploy ingress-controller -n ingress-nginx --name ingress --port=$PORT --target-port=$TARGET_PORT --type NodePort ``` - Question 7 - edit the ingress svc: ``` kubectl edit svc ingress-nginx -n ingress-space ``` # Section 10: Design a Kubernetes Cluster - This is more of a informational section. ## Choosing a Kuberetes Infrastructure - This is more of a informational section. ## Configure High Availability - This is more of a informational section. ## ETCD in HA - Reference [[#ETCD For Beginners]] & [[#ETCD in Kubernetes]] - The recommended quorum is 3 for fault tolerance of 1. - Be mindful of how you segment etcd in the environment as this can lead to quorum failures. Odd number nodes is recommended for quorum. - e.g. you have 6 nodes but 3 in different DCs. This will result in quorum failure due to the nodes not having the required 4 nodes for the quorum to work as expected. ## Update: Kubernetes The Hardway - Youtube training: - [https://www.youtube.com/watch?v=uUupRagM7m0&list=PL2We04F3Y_41jYdadX55fdJplDvgNGENo](https://www.youtube.com/watch?v=uUupRagM7m0&list=PL2We04F3Y_41jYdadX55fdJplDvgNGENo) - The GIT Repo for this tutorial can be found here: - [https://github.com/mmumshad/kubernetes-the-hard-way](https://github.com/mmumshad/kubernetes-the-hard-way) # Section 11: Install "Kubernetes the kubeadm way" - https://github.com/kodekloudhub/certified-kubernetes-administrator-course - Here's the link to the documentation: - https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/ - Encountering too many issues trying to get this working... skipping for now... - Possible solution: https://www.reddit.com/r/pop_os/comments/168mkj7/has_anyone_tried_to_use_vagrant_and_virtualbox_in/ ``` To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Alternatively, if you are the root user, you can run: export KUBECONFIG=/etc/kubernetes/admin.conf You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 192.9.88.6:6443 --token rtwlnn.nyvvnehs5j5eptq7 \ --discovery-token-ca-cert-hash sha256:4126321ec37fce9c38b5ea336e2d9a54a624d1c254d1110314df791c3c194fdd ``` # Section 12: End to End Tests on a Kubernetes Cluster - Not relevant anymore. # Section 13: Troubleshooting ## Application Failure - Basic troubleshooting info such as: - Checking the native app resources, e.g. the app URL & port. - Check the app service: `kubectl desc svc $web-app` - Verify Service to POD discovery. e.g. Do the Selectors match - Check the POD itself: - `kubectl get pod` - `kubectl describe pod $web-app-pod` - `kubectl logs $web-app -f` to follow logs - `kubectl logs $web-app -f --previous` to follow logs & get previous log output - Check dependent services, e.g. DB service - Check the dependent service PODs as described above for the app POD - Reference: https://kubernetes.io/docs/tasks/debug/debug-application/ ### Practice Test - Application Failure - Namespace: alpha - Goal: Application should return green web page when fixed. - Follow-up: edited the **deployment** and got it to work but thats not the solution provided. - solution provided says to edit the mysql service instead. - changed the metadata name from mysql-server to mysql... - Namespace: beta - Goal: App should return green - Solution: Edited the mysql service & updated the port to be 3306 from 8080 - Namespace: gamma - Goal: App should return green - Solution: updated the **selector** to mysql to match the **label** in the pod - Namespace: delta - Goal: App should return green - Solution: Edited **deployment** for the web-app to change sql user to root - Namespace: epsilon - Goal: App should return green - Solution: Edit mysql pod to update password - Namespace: zeta - Goal: App should return green - Solution: - Update web-service svc to use correct nodePort - update deployment to use correct mysql user - update pod to use correct password ## Solution - Application Failure - Set the default namespace: ``` kubectl config set-context --current --namespace=$NAMESPACE ``` ## Control Plane Failures - Reference: https://kubernetes.io/docs/tasks/debug/debug-cluster/ - Obtain debugging info for cluster: ``` kubectl cluster-info dump ``` - Debug a worker node: ``` kubectl describe node kube-worker-1 ``` - Force replace a pod: ``` kubectl replace --force -f $definition.yaml ``` ## Practice Test - Control Plane Failures - Reference upstream cheatsheet. - Don't forget for kube-system pods that they are **static** & the manifests are in: - `/etc/kubernetes/manifests` - Question 2: Scale the PODs to 2 - Solution: Use the **scale** command to increase PODs - `kubectl scale deploy app --replicas=2` - Note: didn't scale up due to error of question 3 - Question 3: Fix kube-controller-manager component - Solution: update static manifest for the service. - Question 4: fix tls cert file location. - Solution: remember that directories are **mounted** to K8S & the paths need to be defined correctly or the files won't be loaded. - Meaning that just because the files exist on the file system, if they aren't defined in a **hostPath** correctly it doesn't matter. ## Worker Node Failures - Check node status as mentioned in [[#Control Plane Failures]] - Basically perform standard Linux system troubleshooting: - Check resources, (CPU, MEM, Disk, etc) - Verify kublet service is running. - Verify kublet TLS certs - CA Issuer is correct - Validity isn't expired ## Practice Test - Worker Node Failures - The service name is **kubelet** - Question: fix node01 - Solution: start the **kubelet** service - Question: fix node01 - Solution: Update kubelet config to reference correct PKI file, `/var/lib/kubelet/config.yaml` - Question 3: fix node01 - Solution: Update kubelet config to reference correct API endpoint. ## Network Troubleshooting - It was basically the following... - Network Plugin in Kubernetes - There are several plugins available and these are some. ### Weave Net To install, ``` kubectl apply -f https://github.com/weaveworks/weave/releases/download/v2.8.1/weave-daemonset-k8s.yaml ``` - You can find details about the network plugins in the following documentation : - https://kubernetes.io/docs/concepts/cluster-administration/addons/#networking-and-network-policy ### Flannel To install, ``` kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.yml ``` Note: As of now flannel does not support kubernetes network policies. ### Calico To install, ``` curl https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/calico.yaml -O ``` - Apply the manifest using the following command. ``` kubectl apply -f calico.yaml ``` - Calico is said to have most advanced cni network plugin. - In CKA and CKAD exam, you won't be asked to install the CNI plugin. But if asked you will be provided with the exact URL to install it. - Note: If there are multiple CNI configuration files in the directory, the kubelet uses the configuration file that comes first by name in lexicographic order. ### DNS in Kubernetes Kubernetes uses CoreDNS. CoreDNS is a flexible, extensible DNS server that can serve as the Kubernetes cluster DNS. ### Memory and Pods - In large scale Kubernetes clusters, CoreDNS's memory usage is predominantly affected by the number of Pods and Services in the cluster. Other factors include the size of the filled DNS answer cache, and the rate of queries received (QPS) per CoreDNS instance. - Kubernetes resources for coreDNS are: - a service account named coredns, - cluster-roles named coredns and kube-dns - clusterrolebindings named coredns and kube-dns, - a deployment named coredns, - a configmap named coredns and a - service named kube-dns. - While analyzing the coreDNS deployment you can see that the the Corefile plugin consists of important configuration which is defined as a configmap. - Port 53 is used for for DNS resolution. ``` kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa ttl 30 } ``` - This is the backend to k8s for cluster.local and reverse domains. ``` proxy . /etc/resolv.conf ``` - Forward out of cluster domains directly to right authoritative DNS server. - Troubleshooting issues related to coreDNS 1. If you find CoreDNS pods in pending state first check network plugin is installed. 2. coredns pods have CrashLoopBackOff or Error state - If you have nodes that are running SELinux with an older version of Docker you might experience a scenario where the coredns pods are not starting. To solve that you can try one of the following options: - a)Upgrade to a newer version of Docker. - b)Disable SELinux. - c)Modify the coredns deployment to set allowPrivilegeEscalation to true: ``` kubectl -n kube-system get deployment coredns -o yaml | \ sed 's/allowPrivilegeEscalation: false/allowPrivilegeEscalation: true/g' | \ kubectl apply -f - ``` - d)Another cause for CoreDNS to have CrashLoopBackOff is when a CoreDNS Pod deployed in Kubernetes detects a loop. - Disable the local DNS cache on host nodes, and restore /etc/resolv.conf to the original. - A quick fix is to edit your Corefile, replacing forward . /etc/resolv.conf with the IP address of your upstream DNS, for example forward . 8.8.8.8. But this only fixes the issue for CoreDNS, kubelet will continue to forward the invalid resolv.conf to all default dnsPolicy Pods, leaving them unable to resolve DNS. 3. If CoreDNS pods and the kube-dns service is working fine, check the kube-dns service has valid endpoints. - `kubectl -n kube-system get ep kube-dns` - If there are no endpoints for the service, inspect the service and make sure it uses the correct selectors and ports. ### Kube-Proxy - kube-proxy is a network proxy that runs on each node in the cluster. kube-proxy maintains network rules on nodes. These network rules allow network communication to the Pods from network sessions inside or outside of the cluster. - In a cluster configured with kubeadm, you can find kube-proxy as a daemonset. - kubeproxy is responsible for watching services and endpoint associated with each service. When the client is going to connect to the service using the virtual IP the kubeproxy is responsible for sending traffic to actual pods. - If you run a kubectl describe ds kube-proxy -n kube-system you can see that the kube-proxy binary runs with following command inside the kube-proxy container. ``` Command: /usr/local/bin/kube-proxy --config=/var/lib/kube-proxy/config.conf --hostname-override=$(NODE_NAME) ``` - So it fetches the configuration from a configuration file ie, /var/lib/kube-proxy/config.conf and we can override the hostname with the node name of at which the pod is running. - In the config file we define the clusterCIDR, kubeproxy mode, ipvs, iptables, bindaddress, kube-config etc. - Troubleshooting issues related to kube-proxy 1. Check kube-proxy pod in the kube-system namespace is running. 2. Check kube-proxy logs. 3. Check configmap is correctly defined and the config file for running kube-proxy binary is correct. 4. kube-config is defined in the config map. 5. check kube-proxy is running inside the container ``` # netstat -plan | grep kube-proxy tcp 0 0 0.0.0.0:30081 0.0.0.0:* LISTEN 1/kube-proxy tcp 0 0 127.0.0.1:10249 0.0.0.0:* LISTEN $ sudo apt install libvirt-daemon-system libvirt-clients qemu-kvm qemu-utils virt-manager ovmf 1/kube-proxy tcp 0 0 172.17.0.12:33706 172.17.0.12:6443 ESTABLISHED 1/kube-proxy tcp6 0 0 :::10256 :::* LISTEN 1/kube-proxy ``` References: - Debug Service issues: - https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/ - DNS Troubleshooting: - https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/ ## Practice Test - Troubleshoot Network - Question 1: Fix app - Solution: install weavenet service - Question 2: fix kube-proxy - Solution: Remember that configurations defined in the definitions are relative to the PODs themselves and not the host. ## Pre-Requisites - JSON PATH - https://kodekloud.com/p/json-path-quiz - Once you are comfortable head back here: - I also have some JSON PATH exercises with Kubernetes Data Objects. Make sure you go through these: - https://mmumshad.github.io/json-path-quiz/index.html#!/?questions=questionskub1 - https://mmumshad.github.io/json-path-quiz/index.html#!/?questions=questionskub2 ## Advanced kubectl Commands ### JSON PATH - Example usage of querying the PODs object using JSON ``` kubectl get pods -o jsonpath='{.items[0].spec.containers[0].image}' ``` - You can query for multiple objects ``` kubectl get pods -o jsonpath='{.items[*].metadata.name} {.items[*].status.capacity.cpu}' ``` - Supports standard escape sequences for newline `{"\n"}` & tab `{"\t"}` - Supports iteration via the `range` keyword ```json '{range .items[*]}{.metadata.name}{"\t"}{.status.capacity.cpu}{"\n"}{end}' ``` - Supports custom colums ``` kubectl get nodes -o=custom-columns=$COL_NAME:$JSON_PATH kubectl get nodes -o=custom-columns=NODE:.metadata.name,CPU:.status.capacity.cpu ``` - Supports sorting ``` kubectl get nodes --sort-by=.metadata.name ``` ## Lightning Lab - 1 - Question 1: update kubeadm - Solution: update repo file to use appropriate version - List available versions: `apt-cache madison kubeadm` - Q2: get output - Solution: ``` kubectl get deployments.apps -n admin2406 -o custom-columns=DEPLOYMENT:.metadata.name,CONTAINER_IMAGE:.spec.template.spec.containers[*].image,READY_REPLICAS:.spec.replicas,NAMESPACE:.metadata.namespace --sort-by=.metadata.name ``` - Q3: - Q4: PVC issue - Solution: pay attention to naming & the read/write option. - Q7: read the secrets volume instead of standard volumes # Section 16: Mock Exams ## Solution - Mock Exam - 1 - Q5: Use the `expose` command to allow access to POD port. - Solution: `kubectl expose pod messaging --port=6379 --name messaging-service` - Q9: don't forget to use the log command when debugging PODs - Q10: Once again use the expose command to provide access to the service. Then edit the service to add the incoming port to listen on. - Solution: ``` k expose deploy hr-web-app --name hr-web-app-service --type NodePort --port 8080 ``` - Q11: Reference the cheat sheet for json parsing. ## Solution - Mock Exam - 2 - Q6-a: CSR - Solution: Create a CSR manifest... not very intuitive on how to do this via docs... - https://kubernetes.io/docs/reference/access-authn-authz/certificate-signing-requests/ - create the request as shown below and then approve it. ``` --- apiVersion: certificates.k8s.io/v1 kind: CertificateSigningRequest metadata: name: john-developer spec: signerName: kubernetes.io/kube-apiserver-client request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ1ZEQ0NBVHdDQVFBd0R6RU5NQXNHQTFVRUF3d0VhbTlvYmpDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRApnZ0VQQURDQ0FRb0NnZ0VCQUt2Um1tQ0h2ZjBrTHNldlF3aWVKSzcrVVdRck04ZGtkdzkyYUJTdG1uUVNhMGFPCjV3c3cwbVZyNkNjcEJFRmVreHk5NUVydkgyTHhqQTNiSHVsTVVub2ZkUU9rbjYra1NNY2o3TzdWYlBld2k2OEIKa3JoM2prRFNuZGFvV1NPWXBKOFg1WUZ5c2ZvNUpxby82YU92czFGcEc3bm5SMG1JYWpySTlNVVFEdTVncGw4bgpjakY0TG4vQ3NEb3o3QXNadEgwcVpwc0dXYVpURTBKOWNrQmswZWhiV2tMeDJUK3pEYzlmaDVIMjZsSE4zbHM4CktiSlRuSnY3WDFsNndCeTN5WUFUSXRNclpUR28wZ2c1QS9uREZ4SXdHcXNlMTdLZDRaa1k3RDJIZ3R4UytkMEMKMTNBeHNVdzQyWVZ6ZzhkYXJzVGRMZzcxQ2NaanRxdS9YSmlyQmxVQ0F3RUFBYUFBTUEwR0NTcUdTSWIzRFFFQgpDd1VBQTRJQkFRQ1VKTnNMelBKczB2czlGTTVpUzJ0akMyaVYvdXptcmwxTGNUTStsbXpSODNsS09uL0NoMTZlClNLNHplRlFtbGF0c0hCOGZBU2ZhQnRaOUJ2UnVlMUZnbHk1b2VuTk5LaW9FMnc3TUx1a0oyODBWRWFxUjN2SSsKNzRiNnduNkhYclJsYVhaM25VMTFQVTlsT3RBSGxQeDNYVWpCVk5QaGhlUlBmR3p3TTRselZuQW5mNm96bEtxSgpvT3RORStlZ2FYWDdvc3BvZmdWZWVqc25Yd0RjZ05pSFFTbDgzSkljUCtjOVBHMDJtNyt0NmpJU3VoRllTVjZtCmlqblNucHBKZWhFUGxPMkFNcmJzU0VpaFB1N294Wm9iZDFtdWF4bWtVa0NoSzZLeGV0RjVEdWhRMi80NEMvSDIKOWk1bnpMMlRST3RndGRJZjAveUF5N05COHlOY3FPR0QKLS0tLS1FTkQgQ0VSVElGSUNBVEUgUkVRVUVTVC0tLS0tCg== usages: - digital signature - key encipherment - client auth ``` - Q6-b: - Create role & rolebinding ``` $ kubectl create role developer --resource=pods --verb=create,list,get,update,delete --namespace=development $ kubectl create rolebinding developer-role-binding --role=developer --user=john --namespace=development ``` - Q7-a: Create nginx pod and a busybox pod to resolve it. - Solution: ``` kubectl expose pod nginx-resolver --name=nginx-resolver-service --port=80 --target-port=80 --type=ClusterIP ``` - Q7-b: - Solution: ``` kubectl run test-nslookup --image=busybox:1.28 --rm -it --restart=Never -- nslookup nginx-resolver-service kubectl run test-nslookup --image=busybox:1.28 --rm -it --restart=Never -- nslookup nginx-resolver-service > /root/CKA/nginx.svc kubectl get pod nginx-resolver -o wide kubectl run test-nslookup --image=busybox:1.28 --rm -it --restart=Never -- nslookup <P-O-D-I-P.default.pod> > /root/CKA/nginx.pod ``` - Q8: Create static pod and copy to node01 ## Mock Exam - 3 - Q1: Create service account, then create a pod under that service account. - Solution: Create the service account via `k create service account $NAME` - Create clusterrole & binding - Create POD with `serviceNameAccount: $NAME` - Q2: Get InternalIP of nodes - solution: `kubectl get nodes -o jsonpath='{.items[*].status.addresses[?(@.type=="InternalIP")].address}'` - Q5: Create network policy - Solution: ```yaml --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: ingress-to-nptest namespace: default spec: podSelector: matchLabels: run: np-test-1 policyTypes: - Ingress ingress: - ports: - protocol: TCP port: 80 ``` - Q6: Create a taints to node01 to be unschedulable & tolerance to schedule to node01