Kubernetes – Basics and Beyond

Copyright © 2022 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Early Access Publication: Kubernetes – Basics and Beyond

Early Access Production Reference: B18998

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK

ISBN: 978-1-80461-139-5

www.packt.com

Table of Contents

Kubernetes – Basics and Beyond, Fourth Edition: Deep dive into Kubernetes and learn how to create and operate world-class container-native systems

1 Understanding Kubernetes Architecture

What is Kubernetes?

What Kubernetes is not?

Understanding container orchestration

Physical machines, virtual machines, and containers

The Benefits of Containers

Containers in the cloud

Cattle versus pets

Kubernetes concepts

Cluster

Node

Control plane

Pod

Label

Label selector

Annotation

Service

Volume

Replication controller and replica set

StatefulSet

Secret

Name

Namespace

Diving into Kubernetes architecture in depth

Distributed systems design patterns

The Kubernetes APIs

Resource Categories

Kubernetes components

Control plane components

Node components

Kubernetes container runtimes

The Container Runtime Interface (CRI)

Docker

Containerd

Cri-O

Mirantis Container Runtime

Lightweight VMs

Summary

Reference

Join our book community on Discord

2 Creating Kubernetes Clusters

Getting ready for your first cluster

Installing Rancher Desktop

Meet Kubectl

Kubectl Alternatives - K9S, KUI and Lens

Creating a single-node cluster with Minikube

Quick Introduction to Minikube

Installing Minikube

Installing Minikube on Windows

Installing Minikube on macOS

Creating the cluster on MacOS

Troubleshooting

Checking out the cluster

Doing work

Examining the cluster with the dashboard

Creating a multi-node cluster with KinD

Quick introduction to KinD

Installing KinD

Dealing with Docker Contexts

Creating the cluster with KinD

Doing work with KinD

Accessing Kubernetes services locally though a proxy

Creating a multi-node cluster with k3d

Quick introduction to k3s and k3d

Installing k3d

Creating the cluster with k3d

Comparing Minikube, KinD and k3d

Honorable Mention - Rancher-Desktop Kubernetes Cluster

Creating clusters in the cloud (GCP, AWS, Azure, Digital Ocean)

The cloud-provider interface

Creating Kubernetes Clusters in the Cloud

GCP

AWS

Azure

Digital Ocean

Other cloud providers

Creating a bare-metal cluster from scratch

Use cases for bare-metal

When should you consider creating a bare-metal cluster?

Understanding the process

Using the Cluster API for managing bare metal clusters

Using virtual private cloud infrastructure

Building your own cluster with Kubespray

Building your cluster with Rancher RKE

Running managed Kubernetes on bare metal or VMs

Summary

References

Join our book community on Discord

Cover

Table of contents

Kubernetes – Basics and Beyond, Fourth Edition: Deep dive into Kubernetes and learn how to create and operate world-class container-native systems

Welcome to Packt Early Access. We’re giving you an exclusive preview of this book before it goes on sale. It can take many months to write a book, but our authors have cutting-edge information to share with you today. Early Access gives you an insight into the latest developments by making chapter drafts available. The chapters may be a little rough around the edges right now, but our authors will update them over time.You can dip in and out of this book or follow along from start to finish; Early Access is designed to be flexible. We hope you enjoy getting to know more about the process of writing a Packt book.

Chapter 1: Understanding Kubernetes Architecture

Chapter 2: Creating Kubernetes Clusters

Chapter 3: High Availability and Reliability

Chapter 4: Securing Kubernetes

Chapter 5: Using Kubernetes Resources in Practice

Chapter 6: Managing Storage

Chapter 7: Running Stateful Applications with Kubernetes

Chapter 8: Deploying and Updating Applications

Chapter 9: Packaging Applications

Chapter 10: Exploring Advanced Networking

Chapter 11: Running Kubernetes on Multiple Clouds and Cluster Federation

Chapter 12: Serverless Computing on Kubernetes

Chapter 13: Monitoring Kubernetes Clusters

Chapter 14: Utilizing Service Meshes

Chapter 15: Extending Kubernetes

Chapter 16: Governing Kubernetes

Chapter 17: Running Kubernetes in Production

Chapter 18: The Future of Kubernetes

1 Understanding Kubernetes Architecture

In one sentence, Kubernetes is a platform to orchestrate the deployment, scaling, and management of container-based applications. You have probably read about Kubernetes, and maybe even dipped your toes in and used it in a side project or maybe even at work. But to understand what Kubernetes is all about, how to use it effectively, and what the best practices are requires much more.Kubernetes is a big open source project and ecosystem with a lot of code and a lot of functionality. Kubernetes came out of Google, but joined the Cloud Native Computing Foundation (CNCF) and is now the de-facto standard in the space of container-based applications. According to the 2021 CNCF survey 96% of organizations use or evaluate Kubernetes.In this chapter, we will build the foundation necessary to utilize Kubernetes to its full potential. We will start by understanding what Kubernetes is, what Kubernetes isn't, and what container orchestration means exactly. Then we will cover important Kubernetes concepts that will form the vocabulary we will use throughout the book. After that, we will dive into the architecture of Kubernetes proper and look at how it enables all the capabilities it provides for its users. Then, we will discuss the how Kubernetes supports multiple container runtimes in a generic way.At the end of this chapter, you will have a solid understanding of container orchestration, what problems Kubernetes addresses, the rationale of Kubernetes design and architecture, and the different runtime engines it supports.

What is Kubernetes?

Kubernetes is a platform that encompasses a huge number of services and capabilities that keeps growing. The core functionality is scheduling workloads in containers across your infrastructure, but it doesn't stop there. Here are some of the other capabilities Kubernetes brings to the table:

Mounting storage systems

Distributing secrets

Checking application health and readiness

Replicating application instances

Using the Horizontal Pod Autoscaler

Using Cluster Autoscaling

Naming and service discovery

Balancing loads

Rolling updates

Monitoring resources

Accessing and ingesting logs

Debugging applications

Providing authentication and authorization

We will cover all these capabilities in great detail throughout the book. At this point, just absorb and appreciate how much value Kubernetes can add to your system.Kubernetes has impressive scope, but it is also important to understand what Kubernetes explicitly doesn't provide.

What Kubernetes is not?

Kubernetes is not a PaaS (platform as a service). It doesn't dictate many important aspects that are left to you or to other systems built on top of Kubernetes such as Deis, OpenShift and Eldarion. For example:

Kubernetes doesn't require specific application type or framework

Kubernetes doesn't require a specific programming language

Kubernetes doesn't provide databases or message queues

Kubernetes doesn't distinguish apps from services

Kubernetes doesn't have a click to deploy service marketplace

Kubernetes doesn't provide a built-in function as a service solution

Kubernetes doesn't mandate a logging, monitoring and alerting systems

Kubernetes doesn't provide a CI/CD pipeline

Understanding container orchestration

The primary responsibility of Kubernetes is container orchestration. That means making sure that all the containers that execute various workloads are scheduled to run on physical or virtual machines. The containers must be packed efficiently following the constraints of the deployment environment and the cluster configuration. In addition, Kubernetes must keep an eye on all running containers and replace dead, unresponsive, or otherwise unhealthy containers. Kubernetes provides many more capabilities that you will learn about in the following chapters. In this section, the focus is on containers and their orchestration.

Physical machines, virtual machines, and containers

It all starts and ends with hardware. In order to run your workloads, you need some real hardware provisioned. That includes actual physical machines, with certain compute capabilities (CPUs or cores), memory, and some local persistent storage (spinning disks or SSDs). In addition, you will need some shared persistent storage and to hook up all these machines using networking, so they can find and talk to each other. At this point, you run multiple virtual machines on the physical machines or stay at the bare-metal level (no virtual machines). Kubernetes can be deployed on a bare-metal cluster (real hardware) or on a cluster of virtual machines. Kubernetes in turn can orchestrate the containers it manages directly on bare-metal or on virtual machines. In theory, a Kubernetes cluster can be composed of a mix of bare-metal and virtual machines, but this is not very common. There are many more esoteric configurations with different level of encapsulation such virtual Kubernetes clusters running inside namespaces of another Kubernetes cluster.

The Benefits of Containers

Containers represent a true paradigm shift in the development and operation of large complicated software systems. Here are some of the benefits compared to more traditional models:

Agile application creation and deployment

Continuous development, integration, and deployment

Dev and Ops separation of concerns

Environmental consistency across development, testing, staging, and production

Cloud and OS distribution portability

Application-centric management

Resource isolation

Resource utilization

Containers in the cloud

Containers are ideal to package microservices because, while providing isolation to the microservice, they are very lightweight and you don't incur a lot of overhead when deploying many microservices as you do with virtual machines. That makes containers ideal for cloud deployment, where allocating a whole virtual machine for each microservice would be cost prohibitive.All major cloud providers, such as Amazon AWS, Google's GCE, Microsoft's Azure provide container hosting services these days. Many other companies jumped on the Kubernetes wagon and offer managed Kubernetes services including: - IBM IKS - Alibaba Alibaba cloud - DigitalOcean DKS - Oracle OKS - OVH Managed Kubernetes - Rackspace KaaSGoogle's GKE was always based on Kubernetes. AWS EKS (Elastic Kubernetes Service) was added in addition to the proprietary AWS ECS orchestration solution. Microsoft Azure's container service used to be based on Apache Mesos, but later switched to Kubernetes with AKS (Azure Kubernetes Service). You could always deploy Kubernetes on all the cloud platforms, but it wasn't deeply integrated with other services. But, at the end of 2017 all cloud providers announced direct support for Kubernetes. Microsoft's launched AKS, AWS released EKS. Also, various other companies offer managed Kubernetes services such as IBM, Oracle, Digital Ocean, Alibaba, Tencent and Huawei.

Cattle versus pets

In the olden days, when systems were small, each server had a name. Developers and users knew exactly what software was running on each machine. I remember that, in many of the companies I worked for, we had multi-day discussions to decide on a naming theme for our servers. For example, composers and Greek mythology characters were popular choices. Everything was very cozy. You treated your servers like beloved pets. When a server died it was a major crisis. Everybody scrambled to try to figure out where to get another server, what was even running on the dead server, and how to get it working on the new server. If the server stored some important data, then hopefully you had an up-to-date backup, and maybe you'd even be able to recover it.Obviously, that approach doesn't scale. When you have tens or hundreds of servers, you must start treating them like cattle. You think about the collective and not individuals. You may still have some pets like your CI/CD machines (although managed CI/CD solutions becoming more common), but your web servers and backend services are just cattle.Kubernetes takes the cattle approach to the extreme and takes full responsibility for allocating containers to specific machines. You don't need to interact with individual machines (nodes) most of the time. This works best for stateless workloads. For stateful applications, the situation is a little different, but Kubernetes provides a solution called StatefulSet, which we'll discuss soon.In this section, we covered the idea of container orchestration and discussed the relationships between hosts (physical or virtual) and containers, as well as the benefits of running containers in the cloud, and finished with a discussion about cattle versus pets. In the following section, we will get to know the world of Kubernetes and learn its concepts and terminology.

Kubernetes concepts

In this section, I'll briefly introduce many important Kubernetes concepts and give you some context as to why they are needed and how they interact with other concepts. The goal is to get familiar with these terms and concepts. Later, we will see how these concepts are woven together and organized into API groups and resource categories to achieve awesomeness. You can consider many of these concepts as building blocks. Some concepts, such as node and master, are implemented as a set of Kubernetes components. These components are at a different abstraction level, and I discuss them in detail in a dedicated section, Kubernetes components.Here is the Kubernetes architecture diagram:

Figure 1.1: Kubernetes architecture

Cluster

A cluster is a collection of hosts (nodes) that provide compute, memory, storage and networking resources. Kubernetes uses these resources to run the various workloads that comprise your system. Note that your entire system may consist of multiple clusters. We will discuss this advanced use case of multi-cluster systems in detail later.

Node

A node is a single host. It may be a physical or virtual machine. Its job is to run pods. Each Kubernetes node runs several Kubernetes components, such as the kubelet, the container runtime and the kube proxy. Nodes are managed by the Kubernetes control plane. The nodes are the worker bees of Kubernetes and shoulder all the heavy lifting. In the past they were called minions. If you read some old documentation or articles, don't get confused. Minions are just nodes.

Control plane

The control plane of Kubernetes consists of several components, such as an API server, a scheduler, a controller manager and optionally a cloud controller manager. The control plane is responsible for the global state of the cluster, cluster-level scheduling of pods and handling of events. Usually, all the control plane components are set up on the same host although its not required. When considering high-availability scenarios or very large clusters, you will want to have control plane redundancy. We will discuss highly available clusters in detail in Chapter 4, Kubernetes High Availability and Scaling.

Pod

A pod is the unit of work in Kubernetes. Each pod contains one or more containers (so you can think of it as a container container). A pods is scheduled as an atomic unit (all its containers run on the same machine). All the containers in a pod have the same IP address and port space; they can communicate with each other using localhost or standard inter-process communication. In addition, all the containers in a pod can have access to shared local storage on the node hosting the pod. Containers don't get access to local storage or any other storage by default. Volumes of storage must be mounted into each container inside the pod explicitly. Pods are important feature of Kubernetes. It is possible to run multiple applications inside a single container by having something like supervisord as the main process that runs multiple processes, but this practice is often frowned upon for the following reasons:

Transparency : Making the containers within the pod visible to the infrastructure enables the infrastructure to provide services to those containers, such as process management and resource monitoring. This facilitates a number of conveniences for users.

Decoupling software dependencies : The individual containers may be versioned, rebuilt, and redeployed independently. Kubernetes may even support live updates of individual containers someday.

Ease of use : Users don't need to run their own process managers, worry about signal and exit-code propagation, and so on.

Efficiency : Because the infrastructure takes on more responsibility, containers can be more lightweight.

Pods provide a great solution for managing groups of closely related containers that depend on each other and need to co-operate on the same host to accomplish their purpose. It's important to remember that pods are considered ephemeral, throwaway entities that can be discarded and replaced at will. Each pod gets a unique ID (UID), so you can still distinguish between them if necessary.

Label

Labels are key-value pairs that are used to group together sets of objects, very often pods via selectors. This is important for several other concepts, such as replica sets, deployments and services that operate on dynamic groups of objects and need to identify the members of the group. There is a NxN relationship between objects and labels. Each object may have multiple labels, and each label may be applied to different objects. There are certain restrictions by design on labels. Each label on an object must have a unique key. The label key must adhere to a strict syntax. Note that labels are dedicated for identifying objects and not for attaching arbitrary metadata to objects. This is what annotations are for (see the following section).

Label selector

Label selectors are used to select objects based on their labels. Equality-based selectors specify a key name and a value. There are two operators, = (or ==) and !=, for equality or inequality based on the value. For example:

role = webserver

This will select all objects that have that label key and value.Label selectors can have multiple requirements separated by a comma. For example:

role = webserver, application != foo

Set-based selectors extend the capabilities and allow selection based on multiple values:

role in (webserver, backend)

Annotation

Annotations let you associate arbitrary metadata with Kubernetes objects. Kubernetes just stores the annotations and makes their metadata available. Annotation key syntax has similar requirements as label keys.In my experience, you always need such metadata for complicated systems, and it is nice that Kubernetes recognizes this need and provides it out of the box, so you don't have to come up with your own separate metadata store and mapping object to their metadata.

Service

Services are used to expose some functionality to users or other services. They usually encompass a group of pods, usually identified by – you guessed it – a label. You can have services that provide access to external resources, or pods you control directly at the virtual IP level. Native Kubernetes services are exposed through convenient endpoints. Note that services operate at layer 3 (TCP/UDP). Kubernetes 1.2 added the Ingress object, which provides access to HTTP objects. More on that later. Services are published or discovered via one of two mechanisms: DNS, or environment variables. Services can be load-balanced inside the cluster by Kubernetes. But, developers can choose to manage load balancing themselves in case of services that use external resources or require special treatment.There are many gory details associated with IP addresses, virtual IP addresses, and port spaces. We will discuss them in depth in a future chapter.

Volume

Local storage used by the pod is ephemeral and goes away with the pod in most cases. Sometimes that's all you need, if the goal is just to exchange data between containers of the node, but sometimes it's important for the data to outlive the pod, or it's necessary to share data between pods. The volume concept supports that need. The essence of a volume is a directory with some data that is mounted into a container.There are many volume types. Originally, Kubernetes supported directly many volume types, but the modern approach for extending Kubernetes with volume types is through the Container Storage Interface (CSI), which we'll discuss in detail later. Most of the originally built-in volume types will have been (or are in the process of being) phased out in favor of out-of-tree plugins available though CSI.

Replication controller and replica set

Replication controllers and replica sets both manage a group of pods identified by a label selector and ensure that a certain number is always up and running. The main difference between them is that replication controllers test for membership by name equality and replica sets can use set-based selection. Replica sets are the way to go as they are a superset of replication controllers. I expect replication controllers to be deprecated at some point. Kubernetes guarantees that you will always have the same number of pods running as you specified in a replication controller or a replica set. Whenever the number drops due to a problem with the hosting node or the pod itself, Kubernetes will fire up new instances. Note that, if you manually start pods and exceed the specified number, the replica set controller will kill some extra pods.Replication controllers used to be central to many workflows, such as rolling updates and running one-off jobs. As Kubernetes evolved, it introduced direct support for many of these workflows, with dedicated objects such as Deployment, Job, CronJob, and DaemonSet. We will meet them all later.

StatefulSet

Pods come and go, and if you care about their data then you can use persistent storage. That's all good. But sometimes you want Kubernetes to manage a distributed data store such as Cassandra or CockroachDB Galera. These clustered stores keep the data distributed across uniquely identified nodes. You can't model that with regular pods and services. Enter StatefulSet. If you remember earlier, we discussed pets versus cattle and how cattle is the way to go. Well, StatefulSet sits somewhere in the middle. StatefulSet ensures (similar to a ReplicaSet) that a given number of instances with unique identities are running at any given time. StatefulSet members have the following properties:

A stable hostname, available in DNS

An ordinal index

Stable storage linked to the ordinal and hostname

StatefulSet can help with peer discovery as well as adding or removing members safely.

Secret

Secrets are small objects that contain sensitive info such as credentials and tokens. They are stored by default as plaintext in etcd, accessible by the Kubernetes API server, and can be mounted as files into pods (using dedicated secret volumes that piggyback on regular data volumes) that need access to them. The same secret can be mounted into multiple pods. Kubernetes itself creates secrets for its components, and you can create your own secrets. Another approach is to use secrets as environment variables. Note that secrets in a pod are always stored in memory (tmpfs in the case of mounted secrets) for better security. The best practice is to enable encryption at rest as well as access control with RBAC. We will discuss it in detail later.

Name

Each object in Kubernetes is identified by a UID and a name. The name is used to refer to the object in API calls. Names should be up to 253 characters long and use lowercase alphanumeric characters, dash (-) and dot (.). If you delete an object, you can create another object with the same name as the deleted object, but the UIDs must be unique across the lifetime of the cluster. The UIDs are generated by Kubernetes, so you don't have to worry about it.

Namespace

A namespace is a form of isolation that let you group resources and apply policies. It is also a scope for names. Objects of the same kind must have unique names within a namespace. By default, pods in one namespace can access pods and services in other namespaces. Note that there cluster-scope objects like Node objects and persistent volumes that don't live in a namespace. Kubernetes may schedule pods from different namespaces to run on the same node. Likewise, pods from different namespaces can use the same persistent storage.In multi-tenancy scenarios where it's important to totally isolate namespaces you can do a passable job with proper network policies and resource quotas to ensure proper access and distribution of the physical cluster resources. But, in general namespaces are considered a weak form of isolation and there are other solutions more appropriated for hard multi-tenancy like virtual clustera, which we will discuss later. We've covered most of Kubernetes' primary concepts; there are a few more I mentioned briefly. In the next section, we will continue our journey into Kubernetes architecture by looking into its design motivations, the internals and implementation, and even pick at the source code.

Diving into Kubernetes architecture in depth

Kubernetes has very ambitious goals. It aims to manage and simplify the orchestration, deployment, and management of distributed systems across a wide range of environments and cloud providers. It provides many capabilities and services that should work across all these diverse environments and use cases, while evolving and remaining simple enough for mere mortals to use. This is a tall order. Kubernetes achieves this by following a crystal-clear, high-level design and well-thought-out architecture that promotes extensibility and pluggability. Kubernetes originally had many hard-coded or environment-aware components, but the trend is to refactor them into plugins and keep the core small, generic and abstract. In this section, we will peel Kubernetes like an onion, starting with various distributed systems design patterns and how Kubernetes supports them, then go over the surface of Kubernetes, which is its set of APIs, and then take a look at the actual components that comprise Kubernetes. Finally, we will take a quick tour of the source-code tree to gain even better insight into the structure of Kubernetes itself.At the end of this section, you will have a solid understanding of Kubernetes architecture and implementation, and why certain design decisions were made.

Distributed systems design patterns

All happy (working) distributed systems are alike, to paraphrase Tolstoy in Anna Karenina. That means that, to function properly, all well-designed distributed systems must follow some best practices and principles. Kubernetes doesn't want to be just a management system. It wants to support and enable these best practices and provide high-level services to developers and administrators. Let's look at some of those described as design patterns.

Sidecar pattern

The sidecar pattern is about co-locating another container in a pod in addition to the main application container. The application container is unaware of the sidecar container and just goes about its business. A great example is a central logging agent. Your main container can just log to stdout, but the sidecar container will send all logs to a central logging service where they will be aggregated with the logs from the entire system. The benefits of using a sidecar container versus adding central logging to the main application container are enormous. First, applications are not burdened anymore with central logging, which could be a nuisance. If you want to upgrade or change your central logging policy or switch to a totally new provider, you just need to update the sidecar container and deploy it. None of your application containers change, so you can't break them by accident. The istio service mesh uses the sidecar pattern to inject its proxies into each pod.

Ambassador pattern

The ambassador pattern is about representing a remote service as if it were local and possibly enforcing some policy. A good example of the ambassador pattern is if you have a Redis cluster with one master for writes and many replicas for reads. A local ambassador container can serve as a proxy and expose Redis to the main application container on the localhost. The main application container simply connects to Redis on localhost:6379 (Redis default port), but it connects to the ambassador running in the same pod, which filters the requests, and sends write requests to the real Redis master and read requests randomly to one of the read replicas. Just like with the sidecar pattern, the main application has no idea what's going on. That can help a lot when testing against a real local Redis. Also, if the Redis cluster configuration changes, only the ambassador needs to be modified; the main application remains blissfully unaware.

Adapter pattern

The adapter pattern is about standardizing output from the main application container. Consider the case of a service that is being rolled out incrementally: it may generate reports in a format that doesn't conform to the previous version. Other services and applications that consume that output haven't been upgraded yet. An adapter container can be deployed in the same pod with the new application container and massage their output to match the old version until all consumers have been upgraded. The adapter container shares the filesystem with the main application container, so it can watch the local filesystem, and whenever the new application writes something, it immediately adapts it.

Multi-node patterns

The single-node patterns are all supported directly by Kubernetes via pods. Multi-node patterns such as leader election, work queues, and scatter-gather are not supported directly, but composing pods with standard interfaces to accomplish them is a viable approach with Kubernetes.

Level-triggered Infrastructure and reconciliation

Kubernetes is all about control loops. It keeps watching itself and correcting issues. Level-triggered infrastructure means that Kubernetes has a desired state, and it constantly strives towards it. For example, if a replica set has a desired state of 3 replicas and it drops to 2 replicas Kubernetes (the ReplicaSet controller part of Kubernetes) will notice and work to get back to 3 replicas. The alternative approach of edge-triggering is - if the number of replicas dropped from 2 to 3 create a new replica. This approach is very brittle and has many edge cases especially in distributed systems where event like replicas coming and going can happen simultaneously.

The Kubernetes APIs

If you want to understand the capabilities of a system and what it provides, you must pay a lot of attention to its API. The API provides a comprehensive view of what you can do with the system as a user. Kubernetes exposes several sets of REST APIs for different purposes and audiences via API groups. Some APIs are used primarily by tools and some can be used directly by developers. An important aspect of the APIs is that they are under constant development. The Kubernetes developers keep it manageable by trying to extend (adding new objects and new fields to existing objects) and avoid renaming or dropping existing objects and fields. In addition, all API endpoints are versioned, and often have an alpha or beta notation too. For example:

/api/v1
/api/v2alpha1

You can access the API through the kubectl CLI, via client libraries, or directly through REST API calls. There are elaborate authentication and authorization mechanism we will explore in a later chapter. If you have the right permissions you can list, view, create, update and delete various Kubernetes objects. At this point, let's get a glimpse into the surface area of the APIs. The best way to explore the API is via API groups. Some API groups are enabled by default. Other groups can be enabled/disabled via flags. For example, to disable the autoscaling/v1 group and enable the autoscaling/v2beta2 group you can set the --runtime-config flag when running the API server as follows:

--runtime-config=autoscaling/v1=false,autoscaling/v2beta2=true

Note that managed Kubernetes clusters in the cloud don't let you specify flags for the API server (as they manage it).

Resource Categories

In addition to API groups, another useful classification of available APIs is by functionality. The Kubernetes API is huge and breaking it down into categories helps a lot when you're trying to find your way around. Kubernetes defines the following resource categories:

Workloads : objects you use to manage and run containers on the cluster.

Discovery and Load Balancing : objects you use to expose your workloads to the world as externally accessible, load-balanced services.

Config and Storage : objects you use to initialize and configure your applications, and to persist data that outside the container.

Cluster : objects that define how the cluster itself is configured; these are typically used only by cluster operators.

Metadata : objects you use to configure the behavior of other resources within the cluster, such as HorizontalPodAutoscaler for scaling workloads.

In the following sub-sections I'll list the resources that belong to each group with the API group they belong to. I will not specify the version here because APIs moves rapidly from alpha to beta to GA (general availability) and from V1 to V2, and so on.

Workloads Resource Category

The workloads category contains the following resources with their corresponding API groups:

Container: core

CronJob: batch

ControllerRevision: apps

DaemonSet: apps

Deployment: apps

HorizontalPodAutoscaler: autoscaling

Job: batch

Pod: core

PodTemplate: core

PriorityClass: scheduling.k8s.io

ReplicaSet: apps

ReplicationController: core

StatefulSet: apps

Containers are created by Controllers through Pods. Pods run Containers and provide environmental dependencies such as shared or persistent storage Volumes and Configuration or Secret data injected into the container.Here is a detailed documentation of one of the most common operations, get a list of all the pods across all namespaces as a REST API:

GET /api/v1/pods

It accepts various query parameters (all optional):

fieldSelector: a selector to restrict the list of returned objects by their fields. Defaults to everything.

labelSelector A selector to restrict the list of returned objects by their labels. Defaults to everything.

limit/continue: limit is a maximum number of responses to return for a list call. If more items exist, the server will set the continue field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results.

pretty: If 'true', then the output is pretty printed. resourceVersion: resourceVersion sets a constraint on what resource versions a request may be served from. Defaults to unset resourceVersionMatch: resourceVersionMatch determines how resourceVersion is applied to list calls. Defaults to unset timeoutSeconds: Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity. watch/allowWatchBookmarks: Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.

Discovery and Load Balancing

By default, Workloads are only accessible within the cluster, and they must be exposed externally using either a LoadBalancer or a NodePort Service. For development, internally accessible Workloads can be accessed via proxy through the API server using the kubectl proxy command.

Endpoints: core

EndpointSlice: discovery.k8s.io/v1

Ingress: networking.k8s.io

IngressClass: networking.k8s.io

Service: core

Config and Storage

Dynamic configuration without redeployment and secret management are a corner stone of Kubernetes and running complex distributed applications on your Kubernetes cluster. The secret and configuration are not baked into container images and are stored in Kubernetes state store (usually etcd). Kubernetes also provides a lot of abstractions for managing arbitrary storage. Here are some of the primary resources:

ConfigMap: core

CSIDriver: storage.k8s.io

CSINode: storage.k8s.io

CSIStorageCapacity: storage.k8s.io

Secret: core

PersistentVolumeClaim: core

StorageClass: storage.k8s.io

Volume: storage.k8s.io

VolumeAttachment: storage.k8s.io

Metadata

The metadata resources typically show up as sub-resources of the resources the configure. For example, a limit range will be part of a pod configuration. You will not interact with these objects directly most of the time. There are many metadata resources. You can find the complete list here: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.15/#-strong-metadata-apis-strong-

Cluster

The resources in the cluster category are designed for use by cluster operators as opposed to developers. There are many resources in this category as well. Here are some of the most important resources:

Namespace: core

Node: core

PersistentVolume: core

ResourceQuota : core

Role: rbac.authorization.k8s.io

RoleBinding: rbac.authorization.k8s.io

ClusterRole: rbac.authorization.k8s.io

ClusterRoleBinding: rbac.authorization.k8s.io

NetworkPolicy: networking.k8s.io

Now, that we understand how Kubernetes organizes and exposes its capabilities via API groups and resource categories let's see how it manages the physical infrastructure and keeps it up with the state of the cluster.

Kubernetes components

A Kubernetes cluster has several control plane components used to control the cluster, as well as node components that run on each worker node. Let's get to know all these components and how they work together.

Control plane components

The control plane components can all run on one node, but in a highly available setup or a very large cluster, they may be spread across multiple nodes.

API server

The kubernetes API server exposes the Kubernetes REST API. It can easily scale horizontally as it is stateless and stores all the data in the etcd cluster (or another data store in Kubernetes distributions like k3s). The API server is the embodiment of the Kubernetes control plane.

Etcd

Etcd is a highly reliable distributed data store. Kubernetes uses it to store the entire cluster state. In small, transient clusters a single instance of etcd can run on the same node with all the other control plane components. But, for more substantial clusters it is typical to have a 3-node or even 5-node etcd cluster for redundancy and high availability.

Kube controller manager

The Kube controller manager is a collection of various managers rolled up into one binary. It contains the replica set controller, the pod controller, the service controller, the endpoints controller, and others. All these managers watch over the state of the cluster via the API and their job is to steer the cluster into the desired state.

Cloud controller manager

When running in the cloud Kubernetes allows cloud providers to integrate their platform for the purpose of managing nodes, routes, services and volumes. The cloud provider code interacts with Kubernetes code. It replaces some of the functionality of the Kube controller manager. When running Kubernetes with a cloud controller manager you must set the Kube controller manager flag --cloud-provider to "external". This will disable the control loops that the cloud controller manager is taking over. The cloud controller manager was introduced in Kubernetes 1.6, and it's being used by multiple cloud providers already such as:

GCP

AWS

Azure

BaiduCloud

Digital Ocean

Oracle

Linode

OK. Let's look at some code. Kubernetes is implemented in Go. A quick note about Go to help you parse the code: The method name comes first, followed by the method's parameters in parentheses. Each parameter is a pair, consisting of a name followed by its type. Finally, the return values are specified. Go allows multiple return types. It is very common to return an error object in addition to the actual result. If everything is OK, the error object will be nil.Here is the main interface of the cloudprovider package:

``` package cloudprovider
import ( "context" "errors" "fmt" "strings"
v1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/types"
"k8s.io/client-go/informers"
clientset "k8s.io/client-go/kubernetes"
restclient "k8s.io/client-go/rest"
)
// Interface is an abstract, pluggable interface for cloud providers. type Interface interface { Initialize(clientBuilder ControllerClientBuilder, stop <-chan struct{}) LoadBalancer() (LoadBalancer, bool) Instances() (Instances, bool) InstancesV2() (InstancesV2, bool) Zones() (Zones, bool) Clusters() (Clusters, bool) Routes() (Routes, bool) ProviderName() string HasClusterID() bool } ```

Most of the methods return other interfaces with their own method. For example, here is the LoadBalancer interface:

type LoadBalancer interface { GetLoadBalancer(ctx context.Context, clusterName string, service *v1.Service) (status *v1.LoadBalancerStatus, exists bool, err error) GetLoadBalancerName(ctx context.Context, clusterName string, service *v1.Service) string EnsureLoadBalancer(ctx context.Context, clusterName string, service *v1.Service, nodes []*v1.Node) (*v1.LoadBalancerStatus, error) UpdateLoadBalancer(ctx context.Context, clusterName string, service *v1.Service, nodes []*v1.Node) error EnsureLoadBalancerDeleted(ctx context.Context, clusterName string, service *v1.Service) error }

Kube Scheduler

The kube-scheduler is responsible for scheduling pods into nodes. This is a very complicated task as it needs to consider multiple interacting factors, such as:

Resource requirements

Service requirements

Hardware/software policy constraints

Node affinity and anti-affinity specifications

Pod affinity and anti-affinity specifications

Taints and tolerations

Local storage requirements

Data locality

Deadlines

If you need some special scheduling logic not covered by the default Kube scheduler you can replace it with your own custom scheduler. You can also run your custom scheduler side by side with the default scheduler and have your custom scheduler schedule only some subset of the pods.

DNS

Starting with Kubernetes 1.3, a DNS service is part of the standard Kubernetes cluster. It is scheduled as a regular pod. Every service (except headless services) receives a DNS name. Pods can receive a DNS name too. This is very useful for automatic discovery.

Node components

Nodes in the cluster need a couple of components to interact with the API server, receive workloads to execute, and update the API server regarding their status.

Proxy

The kube proxy does low-level network housekeeping on each node. It reflects the Kubernetes services locally and can do TCP and UDP forwarding. It finds cluster IPs via environment variables or DNS.

Kubelet

The kubelet is the Kubernetes representative on the node. It oversees communicating with the API server and manages the running pods. That includes the following:

Receive pod specs

Download pod secrets from the API server

Mount volumes

Run the pod's containers (via the configured container runtime)

Report the status of the node and each pod

Run container liveness probes

In this section, we dug into the guts of Kubernetes and explored its architecture from a very high level of vision and supported design patterns, through its APIs and the components used to control and manage the cluster. In the next section, we will take a quick look at the various runtimes that Kubernetes supports.

Kubernetes container runtimes

Kubernetes originally only supported Docker as a container runtime engine. But that is no longer the case. Kubernetes now supports any runtime that implements the CRI interface.In this section, you'll get a closer look at the CRI and get to know some runtime engines that implement it. At the end of this section, you'll be able to make a well-informed decision about which container runtime is appropriate for your use case and under what circumstances you may switch or even combine multiple runtimes in the same system.

The Container Runtime Interface (CRI)

The CRI is a gRPC API, specifications/requirements and libraries for container runtimes to integrate with the kubelet on a node. In Kubernetes 1.7 the internal Docker integration in Kubernetes was replaced with a CRI-based integration. This was a big deal. It opened the door to multiple implementations that can take advantage of advances in the containers world. The Kubelet doesn't need to interface directly with multiple runtimes. Instead, it can talk to any CRI-compliant container runtime. The following diagram illustrates the flow:

Figure 1.2: kubelet and cri

There are two gRPC service interfaces: ImageService and RuntimeService that CRI container runtimes (or shims) must implement. The ImageService is responsible for managing images. Here is the gRPC/protobuf interface (this is not Go):

service ImageService { rpc ListImages(ListImagesRequest) returns (ListImagesResponse) {} rpc ImageStatus(ImageStatusRequest) returns (ImageStatusResponse) {} rpc PullImage(PullImageRequest) returns (PullImageResponse) {} rpc RemoveImage(RemoveImageRequest) returns (RemoveImageResponse) {} rpc ImageFsInfo(ImageFsInfoRequest) returns (ImageFsInfoResponse) {} }

The RuntimeService is responsible for managing pods and container. Here is the gRPC/protobuf interface:

service RuntimeService { rpc Version(VersionRequest) returns (VersionResponse) {} rpc RunPodSandbox(RunPodSandboxRequest) returns (RunPodSandboxResponse) {} rpc StopPodSandbox(StopPodSandboxRequest) returns (StopPodSandboxResponse) {} rpc RemovePodSandbox(RemovePodSandboxRequest) returns (RemovePodSandboxResponse) {} rpc PodSandboxStatus(PodSandboxStatusRequest) returns (PodSandboxStatusResponse) {} rpc ListPodSandbox(ListPodSandboxRequest) returns (ListPodSandboxResponse) {} rpc CreateContainer(CreateContainerRequest) returns (CreateContainerResponse) {} rpc StartContainer(StartContainerRequest) returns (StartContainerResponse) {} rpc StopContainer(StopContainerRequest) returns (StopContainerResponse) {} rpc RemoveContainer(RemoveContainerRequest) returns (RemoveContainerResponse) {} rpc ListContainers(ListContainersRequest) returns (ListContainersResponse) {} rpc ContainerStatus(ContainerStatusRequest) returns (ContainerStatusResponse) {} rpc UpdateContainerResources(UpdateContainerResourcesRequest) returns (UpdateContainerResourcesResponse) {} rpc ExecSync(ExecSyncRequest) returns (ExecSyncResponse) {} rpc Exec(ExecRequest) returns (ExecResponse) {} rpc Attach(AttachRequest) returns (AttachResponse) {} rpc PortForward(PortForwardRequest) returns (PortForwardResponse) {} rpc ContainerStats(ContainerStatsRequest) returns (ContainerStatsResponse) {} rpc ListContainerStats(ListContainerStatsRequest) returns (ListContainerStatsResponse) {} rpc UpdateRuntimeConfig(UpdateRuntimeConfigRequest) returns (UpdateRuntimeConfigResponse) {} rpc Status(StatusRequest) returns (StatusResponse) {} }

The data types used as arguments and return types are called messages and are also defined in as part of the API. Here is one of them:

message CreateContainerRequest { string pod_sandbox_id = 1; ContainerConfig config = 2; PodSandboxConfig sandbox_config = 3; }

As you can see messages can be embedded inside each other. The CreateContainerRequest message has one string field and two other fields, which are themselves messages: ContainerConfig and PodSandboxConfig.Now that you are familiar at the code level with what Kubernetes considers as a runtime engine, let's look at the individual runtime engines briefly.

Docker

Docker used to be the 800 pound gorilla of containers. Kubernetes was originally designed to manage only Docker containers. The multi-runtime capability was first introduced in Kubernetes 1.3 and the CRI in Kubernetes 1.5. Until then, Kubernetes could only manage Docker containers. Even after CRI was introduced a Dockershim remained in the Kubernetes source code and it was only removed in Kubernetes 1.24. Since then, Docker doesn't get any special treatment anymore.I assume you're very familiar with Docker and what it brings to the table if you are reading this book. Docker enjoys tremendous popularity and growth, but there is also a lot of criticism towards it. Critics often mention the following concerns:

Security

Difficulty setting up multi-container applications (in particular, networking)

Development, monitoring, and logging

Limitations of Docker containers running one command

Releasing half-baked features too fast

Docker is aware of the criticisms and has addressed some of these concerns. In particular, Docker invested in its Docker swarm product. Docker swarm is a Docker-native orchestration solution that competes with Kubernetes. It is simpler to use than Kubernetes, but it's not as powerful or mature.Starting with Docker 1.12, swarm mode is included in the Docker daemon natively, which upset some people due to bloat and scope creep. As a result more people turned to other container runtimes.Starting with Docker 1.11, released on April 2016, Docker has changed the way it runs containers. The runtime now uses containerd and runC to run Open Container Initiative ( OCI ) images in containers:

Figure 1.3: docker and oci

In September 2021 Docker require a paid subscription of Docker Desktop for large organizations. This wasn't popular as you might expect and lots of organizations scrambled to find an alternative. Docker Desktop is client-side distribution and UI for docker and doesn't impact the container runtime. But, it eroded Docker's reputation and goodwill with the community even further.

Containerd

Containerd is a graduated CNCF project since 2019. It is now a mainstream option for Kubernetes containers. All major cloud providers support it and it will soon become the default.In addition the Docker container runtime is built on top of containerd as well.

Cri-O

Cri-O is a CNCF incubator project. It is designed to provide an integration path between Kubernetes and OCI-compliant container runtimes like Docker. Cri-O provides the following capabilities:

Support multiple image formats including the existing Docker image format

Support for multiple means to download images including trust & image verification

Container image management (managing image layers, overlay filesystems, etc)

Container process lifecycle management

Monitoring and logging required to satisfy the CRI

Resource isolation as required by the CRI

It supports runc and Kata containers right now, but any OCI-compliant container runtime can be plugged in and be integrated with Kubernetes.

Mirantis Container Runtime

At the end of 2019 Mirantis acquired the Docker enterprise platform the Docker Inc. It rebranded it as the Mirantis Container Runtime. It provides various enterprise features and supports legacy systems. It might be interesting

Lightweight VMs

Kubernetes runs containers from different applications on the same node, sharing the same OS. This allows running a lot of containers in a very efficient manner. However, container isolation is a serious security concern and multiple cases of privilege escalation occured, which drove a lot of interest with a different approach. Lightweight VMs provide strong VM-level isolation, but are not as heavyweight as standard VMs, which allow them to operate as container runtime on Kubernetes. Some prominent projects are:

AWS Firecracker

Google gVisor

Kata Containers

Singularity

SmartOS

In this section, we've covered the various runtime engines that Kubernetes supports as well as the trend toward standardization, convergence and externalizing the runtime support from core Kubernetes.

Summary

In this chapter, we covered a lot of ground. You learned about the organization, design and architecture of Kubernetes. Kubernetes is an orchestration platform for microservice-based applications running as containers. Kubernetes clusters have a control plane and worker nodes. Containers run within pods. Each pod runs on a single physical or virtual machine. Kubernetes directly supports many concepts, such as services, labels, and persistent storage. You can implement various distributed systems design patterns on Kubernetes. Container runtimes just need to implement the CRI. Docker, containerd, CRI-O and more are supported.In Chapter 2, Creating Kubernetes Clusters, we will explore the various ways to create Kubernetes clusters, discuss when to use different options, and build a local multi-node cluster.

Reference

https://www.cncf.io/reports/cncf-annual-survey-2021/https://kubernetes.io/docs/reference/using-api/api-conceptshttps://kubernetes.io/dockershimhttps://github.com/kubernetes/fraktihttps://github.com/openstack/stackube https://cd.foundation/projects/https://www.huaweicloud.com/intl/en-us/product/cce.htmlhttps://www.alibabacloud.com/help/en/container-service-for-kuberneteshttps://intl.cloud.tencent.com/products/tkehttps://kubernetes.io/docs/setup/production-environment/container-runtimes/https://containerd.io

Join our book community on Discord

https://packt.link/K8s

2 Creating Kubernetes Clusters

In the previous chapter, we learned what Kubernetes is all about, how it is designed, what concepts it supports, its architecture, and the various container runtimes it supports.Creating a Kubernetes cluster from scratch is a non-trivial task. There are many options and tools to select from. There are many factors to consider. In this chapter, we will roll our sleeves and build us some Kubernetes clusters using minikube, kind and k3d. We will discuss and evaluate other tools such as Kubeadm and Kube-spray. We will also look into deployment environments such as local, cloud, and bare metal. The topics we will cover are as follows:

Getting ready for your first cluster

Creating a single-node cluster with Minikube

Creating a multi-node cluster with Kind

Creating a multi-node cluster using k3d

Creating clusters in the cloud

Creating bare-metal clusters from scratch

Reviewing other options for creating Kubernetes clusters

At the end of this chapter, you will have a solid understanding of the various options to create Kubernetes clusters and knowledge of the best-of-breed tools to support the creation of Kubernetes clusters, and you will also build several clusters, both single-node and multi-node.

Getting ready for your first cluster

Before we start creating clusters we should install a couple of tools such as the docker client and kubectl. These days the most convenient way to install docker and kubectl on Mac and Windows is via Rancher Desktop. If you already have them installed feel free to skip this section

Installing Rancher Desktop

Rancher Desktop is Just follow the instructions here: https://docs.rancherdesktop.io/getting-started/installation/Let's verify docker was installed correctly. Type the following commands and make sure you don't see any errors (the output doesn't have to be identical if you installed a different version than mine):

``` $ docker version Client: Version: 20.10.9 API version: 1.41 Go version: go1.16.8 Git commit: c2ea9bc Built: Thu Nov 18 21:17:06 2021 OS/Arch: darwin/arm64 Context: rancher-desktop Experimental: true
Server: Engine: Version: 20.10.14 API version: 1.41 (minimum version 1.12) Go version: go1.17.9 Git commit: 87a90dc786bda134c9eb02adbae2c6a7342fb7f6 Built: Fri Apr 15 00:05:05 2022 OS/Arch: linux/arm64 Experimental: false containerd: Version: v1.5.11 GitCommit: 3df54a852345ae127d1fa3092b95168e4a88e2f8 runc: Version: 1.0.2 GitCommit: 52b36a2dd837e8462de8e01458bf02cf9eea47dd docker-init: Version: 0.19.0 GitCommit: ```

And, while we're at it let's verify kubectl has been installed corredctly too:

$ kubectl version Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.4", GitCommit:"e6c093d87ea4cbb530a7b2ae91e54c0842d8308a", GitTreeState:"clean", BuildDate:"2022-02-16T12:38:05Z", GoVersion:"go1.17.7", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.6+k3s1", GitCommit:"418c3fa858b69b12b9cefbcff0526f666a6236b9", GitTreeState:"clean", BuildDate:"2022-04-28T22:16:58Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/arm64"}

Meet Kubectl

Before we start creating clusters let's talk about kubectl. It is the official Kubernetes CLI, and it interacts with your Kubernetes cluster's API server via its API. It is configured by default using the ~/.kube/config file, which is a YAML file that contains metadata, connection info and authentication tokens or certificates for one or more clusters. Kubectl provides commands to view your configuration and switch between clusters if it contains more than one. You can also point kubectl at a different config file by setting the KUBECONFIG environment variable or passing the --kubeconfig command-line flag.Here is running a kubectl command to check the pods in the kube-system namespace of the current active cluster:

$ kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE svclb-traefik-fv84n 2/2 Running 6 (7d20h ago) 8d local-path-provisioner-84bb864455-s2xmp 1/1 Running 20 (7d20h ago) 27d metrics-server-ff9dbcb6c-lsffr 0/1 Running 88 (10h ago) 27d coredns-d76bd69b-mc6cn 1/1 Running 11 (22h ago) 8d traefik-df4ff85d6-2fskv 1/1 Running 7 (3d ago) 8d

Kubectl Alternatives - K9S, KUI and Lens

Kubectl is no-nonsense command-line tool. It is very powerful, but it may be difficult or less convenient for some people to visually parse its output or remember all the flags and options. There many tools the community developed that can replace (or more like complement) kubectl. The best ones in my opinion are: K9S, KUI and Lens.

K9S

K9S is a terminal-based UI for managing Kubernetes clusters. It has a lot of shortcuts and aggregated views that will require multiple kubectl commands to accomplish.Here is what the k9s window looks like:

Figure 2.1 - k9s

Check it out here: https://k9scli.io

KUI

KUI is a framework for adding graphics to CLIs (command-line interface). This is a very interesting concept. KUI is focused on Kubernetes of course. It lets you run Kubectl commands and returns the results as graphics. KUI also collects a lot of relevant information and presents it in a concise way with tabs and detail pains to explore even deeper.KUI is based on Electron, but it is fastHere is what the KUI window looks like:

Figure 2.2: kui

Lens

Lens is a very polished application. It also presents a graphical view of cluster and allow you to perform a lot of operations from the UI and drop to a terminal interface when necessary. I especially appreciate the ability to work easily with multiple clusters that Lens provides.Here is what the Lens window looks like:

Figure 2.3: lens

Check it out here: https://k8slens.devAll these tools are running locally. I highly recommend that after you play with kubectl a little you give these tools a test drive. One of them may just be your speed.

Creating a single-node cluster with Minikube

In this section, we will create a local single-node cluster using Minikube. Local clusters are most useful for developers that want quick edit-test-deploy-debug cycles on their machine before committing their changes. Local clusters are also very useful for DevOps and operators that want to play with Kubernetes locally without concerns about breaking a shared environment or creating expensive resources in the cloud and forgetting to clean them up. While Kubernetes is typically deployed on Linux in production, many developers work on Windows PCs or Macs. That said, there aren't too many differences if you do want to install Minikube on Linux.

Figure 2.4: minikube

Quick Introduction to Minikube

Minikube is the most mature local Kubernetes cluster. It runs the latest stable Kubernetes release. It supports Windows, macOS and Linux. Minkube provides a lot of advanced options and capabilities:

LoadBalancer service type - via minikube tunnel

NodePort service type - via minikube service

Multiple clusters

Filesystem mounts

GPU support - for machine learning

RBAC

Persistent Volumes

Ingress

Dashboard - via minikube dashboard

Custom container runtimes - via the start --container-runtime flag

Configuring API server and kubelet options via command-line flags

Add-ons

Installing Minikube

The ultimate guide is here: https://minikube.sigs.k8s.io/docs/start/But, to save you a trip here are the latest instructions at the time of writing

Installing Minikube on Windows

On Windows I prefer to install software via the Chocolaty package manager. If you don't have it yet you can get it here: https://chocolatey.org/If you don't want to use Chocolaty check the ultimate guide above for alternative methods.With chocolaty installed the installation is pretty simple:

``` PS C:\Windows\system32> choco install minikube -y Chocolatey v0.12.1 Installing the following packages: minikube By installing, you accept licenses for the packages. Progress: Downloading Minikube 1.25.2... 100%
kubernetes-cli v1.24.0 [Approved] kubernetes-cli package files install completed. Performing other installation steps. Extracting 64-bit C:\ProgramData\chocolatey\lib\kubernetes-cli\tools\kubernetes-client-windows-amd64.tar.gz to C:\ProgramData\chocolatey\lib\kubernetes-cli\tools... C:\ProgramData\chocolatey\lib\kubernetes-cli\tools Extracting 64-bit C:\ProgramData\chocolatey\lib\kubernetes-cli\tools\kubernetes-client-windows-amd64.tar to C:\ProgramData\chocolatey\lib\kubernetes-cli\tools... C:\ProgramData\chocolatey\lib\kubernetes-cli\tools ShimGen has successfully created a shim for kubectl-convert.exe ShimGen has successfully created a shim for kubectl.exe The install of kubernetes-cli was successful. Software installed to 'C:\ProgramData\chocolatey\lib\kubernetes-cli\tools'
Minikube v1.25.2 [Approved] minikube package files install completed. Performing other installation steps. ShimGen has successfully created a shim for minikube.exe The install of minikube was successful. Software installed to 'C:\ProgramData\chocolatey\lib\Minikube'
Chocolatey installed 2/2 packages. See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log). ```

On Windows you can work in different command-line environments. The most common ones are PowerShell and WSL (Windows System for Linux). Eithe one works. You may need to run them in Administrator mode for certaqin operations.As far as console windows go I recommend the official Windows Terminal these days. You can install with one command: choco install microsoft-windows-terminal --preIf you prefer other console windows such as ConEMU or Cmdr this is totally fine.I'll use shortcuts to make life easy. If you want to follow along and copy the aliases into your profile you can use the following:For PowerShell, add the following to your $profile: function k { kubectl.exe $args } function mk { minikube.exe $args }For WSL, add the following to .bashrc:

alias k='kubectl.exe' alias mk=minikube.exe'

Let's verify minikube was installed correctly:

$ mk version minikube version: v1.25.2 commit: 362d5fdc0a3dbee389b3d3f1034e8023e72bd3a7

Let's create a cluster with mk start

``` $ mk start 😄 minikube v1.25.2 on Microsoft Windows 10 Pro 10.0.19044 Build 19044 ✨ Automatically selected the docker driver. Other choices: hyperv, ssh 👍 Starting control plane node minikube in cluster minikube 🚜 Pulling base image ... 💾 Downloading Kubernetes v1.23.3 preload ... > preloaded-images-k8s-v17-v1...: 505.68 MiB / 505.68 MiB 100.00% 3.58 MiB > gcr.io/k8s-minikube/kicbase: 379.06 MiB / 379.06 MiB 100.00% 2.61 MiB p/ 🔥 Creating docker container (CPUs=2, Memory=8100MB) ... 🤦 StartHost failed, but will try again: creating host: create: creating: setting up container node: preparing volume f or minikube container: docker run --rm --name minikube-preload-sidecar --label created_by.minikube.sigs.k8s.io=true --la bel name.minikube.sigs.k8s.io=minikube --entrypoint /usr/bin/test -v minikube:/var gcr.io/k8s-minikube/kicbase:v0.0.30@s ha256:02c921df998f95e849058af14de7045efc3954d90320967418a0d1f182bbc0b2 -d /var/lib: exit status 125 stdout:
stderr: Unable to find image 'gcr.io/k8s-minikube/kicbase:v0.0.30@sha256:02c921df998f95e849058af14de7045efc3954d90320967418a0d1f 182bbc0b2' locally docker: Error response from daemon: Get "https://gcr.io/v2/": net/http: TLS handshake timeout. See 'docker run --help'.
🤷 docker "minikube" container is missing, will recreate. 🔥 Creating docker container (CPUs=2, Memory=8100MB) ... 😿 Failed to start docker container. Running "minikube delete" may fix it: recreate: creating host: create: creating: s etting up container node: preparing volume for minikube container: docker run --rm --name minikube-preload-sidecar --lab el created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --entrypoint /usr/bin/test -v minikub e:/var gcr.io/k8s-minikube/kicbase:v0.0.30@sha256:02c921df998f95e849058af14de7045efc3954d90320967418a0d1f182bbc0b2 -d /v ar/lib: exit status 125 stdout:
stderr: Unable to find image 'gcr.io/k8s-minikube/kicbase:v0.0.30@sha256:02c921df998f95e849058af14de7045efc3954d90320967418a0d1f 182bbc0b2' locally docker: Error response from daemon: Get "https://gcr.io/v2/": net/http: TLS handshake timeout.
❗ Startup with docker driver failed, trying with alternate driver hyperv: Failed to start host: recreate: creating host create: creating: setting up container node: preparing volume for minikube container: docker run --rm --name minikube- : create: creating: setting up container node: preparing volume for minikube container: docker run --rm --name minikube- preload-sidecar --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --entrypoint /us r/bin/test -v minikube:/var gcr.io/k8s-minikube/kicbase:v0.0.30@sha256:02c921df998f95e849058af14de7045efc3954d9032096741 8a0d1f182bbc0b2 -d /var/lib: exit status 125 stdout:
stderr: Unable to find image 'gcr.io/k8s-minikube/kicbase:v0.0.30@sha256:02c921df998f95e849058af14de7045efc3954d90320967418a0d1f 182bbc0b2' locally docker: Error response from daemon: Get "https://gcr.io/v2/": net/http: TLS handshake timeout. See 'docker run --help'.
🔥 Deleting "minikube" in docker ... 🔥 Removing C:\Users\the_g.minikube\machines\minikube ... ❗ Failed to delete cluster minikube, proceeding with retry anyway. 💿 Downloading VM boot image ... > minikube-v1.25.2.iso.sha256: 65 B / 65 B [-------------] 100.00% ? p/s 0s > minikube-v1.25.2.iso: 237.06 MiB / 237.06 MiB [ 100.00% 12.51 MiB p/s 19s 👍 Starting control plane node minikube in cluster minikube 🔥 Creating hyperv VM (CPUs=2, Memory=6000MB, Disk=20000MB) ... ❗ This VM is having trouble accessing https://k8s.gcr.io 💡 To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networki ng/proxy/ 🐳 Preparing Kubernetes v1.23.3 on Docker 20.10.12 ... ▪ kubelet.housekeeping-interval=5m ▪ Generating certificates and keys ... ▪ Booting up control plane ... ▪ Configuring RBAC rules ... 🔎 Verifying Kubernetes components... ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5 🌟 Enabled addons: storage-provisioner, default-storageclass 🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default ```

As you can see the process is pretty complicated even for the default setup and required multiple retries (automatically). You can customize the cluster creation process with a multitude of command-line flags. Type mk start -h to see what's available.Let's check the status of our cluster:

$ mk status minikube type: Control Plane host: Running kubelet: Running apiserver: Running kubeconfig: Configured

All is well!Now let's stop the cluster and later restart it:

gigi@DESKTOP-0SPEBFH:/mnt/c/Users/the_g/git/mastering-kubernetes-4th-edition$ mk stop ✋ Stopping node "minikube" ... 🛑 Powering off "minikube" via SSH ... 🛑 1 node stopped.

Restarting with the time command to measure how long it takes...

``` $ time mk start 😄 minikube v1.25.2 on Microsoft Windows 10 Pro 10.0.19044 Build 19044 ✨ Using the hyperv driver based on existing profile 👍 Starting control plane node minikube in cluster minikube 🔄 Restarting existing hyperv VM for "minikube" ... ❗ This VM is having trouble accessing https://k8s.gcr.io 💡 To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networki ng/proxy/ 🐳 Preparing Kubernetes v1.23.3 on Docker 20.10.12 ... ▪ kubelet.housekeeping-interval=5m 🔎 Verifying Kubernetes components... ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5 🌟 Enabled addons: storage-provisioner, default-storageclass 🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
real 1m8.666s user 0m0.004s sys 0m0.000s ```

It took a little over a minute.Let's review what Minikube did behind the curtains for you. You'll need to do a lot of it when creating a cluster from scratch:

Start a hyperv VM

Create certificates for the local machine and the VM

Download images

Set up networking between the local machine and the VM

Run the local Kubernetes cluster on the VM

Configure the cluster

Start all the Kubernetes control plane components

Configure the kubelet

Enable addons (for storage)

Configure kubectl to talk to the cluster

Installing Minikube on macOS

On Mac I recommend installing minikube using homebrew

``` $ brew install minikube Running brew update --preinstall... ==> Auto-updated Homebrew! Updated 2 taps (homebrew/core and homebrew/cask). ==> Updated Formulae Updated 39 formulae. ==> New Casks contour hdfview rancher-desktop | kube-system ==> Updated Casks Updated 17 casks.
==> Downloading https://ghcr.io/v2/homebrew/core/kubernetes-cli/manifests/1.24.0
100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/kubernetes-cli/blobs/sha256:e57f8f7ea19d22748d1bcae5cd02b91e71816147712e6dcd ==> Downloading from https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:e57f8f7ea19d22748d1bcae5cd02b91e71816147
100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/minikube/manifests/1.25.2 Already downloaded: /Users/gigi.sayfan/Library/Caches/Homebrew/downloads/fa0034afe1330adad087a8b3dc9ac4917982d248b08a4df4cbc52ce01d5eabff--minikube-1.25.2.bottlemanifest.json ==> Downloading https://ghcr.io/v2/homebrew/core/minikube/blobs/sha256:6dee5f22e08636346258f4a6daa646e9102e384ceb63f33981745d Already downloaded: /Users/gigi.sayfan/Library/Caches/Homebrew/downloads/ceeab562206fd08fd3b6523a85b246d48d804b2cd678d76cbae4968d97b5df1f--minikube--1.25.2.arm64monterey.bottle.tar.gz ==> Installing dependencies for minikube: kubernetes-cli ==> Installing minikube dependency: kubernetes-cli ==> Pouring kubernetes-cli--1.24.0.arm64monterey.bottle.tar.gz 🍺 /opt/homebrew/Cellar/kubernetes-cli/1.24.0: 228 files, 55.3MB ==> Installing minikube ==> Pouring minikube--1.25.2.arm64monterey.bottle.tar.gz ==> Caveats zsh completions have been installed to: /opt/homebrew/share/zsh/site-functions ==> Summary 🍺 /opt/homebrew/Cellar/minikube/1.25.2: 9 files, 70.3MB ==> Running brew cleanup minikube... Disable this behaviour by setting HOMEBREWNOINSTALLCLEANUP. Hide these hints with HOMEBREWNOENVHINTS (see man brew). ==> Caveats ==> minikube zsh completions have been installed to: /opt/homebrew/share/zsh/site-functions ```

You can add aliases to your .bashrc file (similar to the WSL aliases on Windows):

alias k='kubectl' alias mk='$(brew --prefix)/bin/minikube'

Now you can use k and mk and type less.Type mk version to verify Minikube is correctly installed and functioning:

$ mk version minikube version: v1.25.2 commit: 362d5fdc0a3dbee389b3d3f1034e8023e72bd3a7

Type k version to verify kubectl is correctly installed and functioning:

$ k version I0522 15:41:13.663004 68055 versioner.go:58] invalid configuration: no configuration has been provided Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.4", GitCommit:"e6c093d87ea4cbb530a7b2ae91e54c0842d8308a", GitTreeState:"clean", BuildDate:"2022-02-16T12:38:05Z", GoVersion:"go1.17.7", Compiler:"gc", Platform:"darwin/amd64"} The connection to the server localhost:8080 was refused - did you specify the right host or port?

Note that the client version is 1.23. Don't worry about the error message. There is no cluster running, so kubectl can't connect to anything. That's expected. We will rectify this in the next section.You can explore the available commands and flags for both Minikube and kubectl. I will not go over each and every one, only the commands I use.

Creating the cluster on MacOS

This is similar on the Mac just run mk start.

Troubleshooting

If something goes wrong during the process, try to follow the error messages. You can add the --alsologtostderr flag to get detailed error info to the console. Everything minikube does is organized neatly under ~/.minikube. Here is the directory structure:

``` $ tree ~/.minikube\ -L 2 C:\Users\the_g.minikube\ |-- addons |-- ca.crt |-- ca.key |-- ca.pem |-- cache | |-- iso | |-- kic | -- preloaded-tarball |-- cert.pem |-- certs | |-- ca-key.pem | |-- ca.pem | |-- cert.pem |-- key.pem |-- config |-- files |-- key.pem |-- logs | |-- audit.json | -- lastStart.txt |-- machine_client.lock |-- machines | |-- minikube | |-- server-key.pem |-- server.pem |-- profiles | -- minikube |-- proxy-client-ca.crt-- proxy-client-ca.key
13 directories, 16 files ```

If you don't have the tree utility you can install it.On Windows:

$ choco install -y tree

On Mac: brew install tree

Checking out the cluster

Now that we have a cluster up and running, let's peek inside.First, let's ssh into the VM:

``` $ mk ssh _ _ _ _ ( ) ( ) ___ ___ () ___ ()| |/') _ _ | |_ __ /' _ _\| |/' _ \| || , < ( ) ( )| '_\ /'\ | ( ) ( ) || || ( ) || || |\\ | () || |) )( / () () ()()() ()()() ()\___/'(_,__/'____)
$ uname -a Linux minikube 4.19.202 #1 SMP Tue Feb 8 19:13:02 UTC 2022 x86_64 GNU/Linux $ ```

Great! That works. The weird marks symbols are ascii art for 'minikube'. Now, let's start using kubectl, because it is the Swiss Army Knife of Kubernetes and will be useful for all clusters.Disconnect from the VM via ctrl+D or typing:

$ logout

We will cover many of the kubectl commands in our journey. First, let's check the cluster status using cluster-info:

``` $ k cluster-info Kubernetes control plane is running at https://172.26.246.89:8443 CoreDNS is running at https://172.26.246.89:8443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. ```

You can see that the control plane is running properly. To see much more detailed view of all the objects in the cluster as JSON type: k cluster-info dump. The output can be a little daunting let's use more specific commands to explore the cluster.Let's check out the nodes in the cluster using get nodes.

$ k get nodes NAME STATUS ROLES AGE VERSION minikube Ready control-plane,master 62m v1.23.3

So, we have one node called minikube. To get a lot more information about it, type

k describe node minikube.

The output is verbose; I'll let you try it yourself.Before we start putting our cluster to work let's check the addons minikube instaled by default:

mk addons list |-----------------------------|----------|--------------|--------------------------------| | ADDON NAME | PROFILE | STATUS | MAINTAINER | |-----------------------------|----------|--------------|--------------------------------| | ambassador | minikube | disabled | third-party (ambassador) | | auto-pause | minikube | disabled | google | | csi-hostpath-driver | minikube | disabled | kubernetes | | dashboard | minikube | disabled | kubernetes | | default-storageclass | minikube | enabled ✅ | kubernetes | | efk | minikube | disabled | third-party (elastic) | | freshpod | minikube | disabled | google | | gcp-auth | minikube | disabled | google | | gvisor | minikube | disabled | google | | helm-tiller | minikube | disabled | third-party (helm) | | ingress | minikube | disabled | unknown (third-party) | | ingress-dns | minikube | disabled | google | | istio | minikube | disabled | third-party (istio) | | istio-provisioner | minikube | disabled | third-party (istio) | | kong | minikube | disabled | third-party (Kong HQ) | | kubevirt | minikube | disabled | third-party (kubevirt) | | logviewer | minikube | disabled | unknown (third-party) | | metallb | minikube | disabled | third-party (metallb) | | metrics-server | minikube | disabled | kubernetes | | nvidia-driver-installer | minikube | disabled | google | | nvidia-gpu-device-plugin | minikube | disabled | third-party (nvidia) | | olm | minikube | disabled | third-party (operator | | | | | framework) | | pod-security-policy | minikube | disabled | unknown (third-party) | | portainer | minikube | disabled | portainer.io | | registry | minikube | disabled | google | | registry-aliases | minikube | disabled | unknown (third-party) | | registry-creds | minikube | disabled | third-party (upmc enterprises) | | storage-provisioner | minikube | enabled ✅ | google | | storage-provisioner-gluster | minikube | disabled | unknown (third-party) | | volumesnapshots | minikube | disabled | kubernetes | |-----------------------------|----------|--------------|--------------------------------|

As you can see minikub comes loaded with a lot of addons, but only enables a couple of storage addons out of the box.

Doing work

Before we start if you have a VPN running, you may need to shut it down when pulling images.We have a nice empty cluster up and running (well, not completely empty as the DNS service and dashboard run as pods in the kube-system namespace). It's time to deploy some pods:

$ k create deployment echo --image=k8s.gcr.io/e2e-test-images/echoserver:2.5 deployment.apps/echo created

Let's check out the pod that was created. The -w flag means watch. Whenever the status changes a new line will be displayed.

$ k get po -w NAME READY STATUS RESTARTS AGE echo-7fd7648898-6hh48 0/1 ContainerCreating 0 5s echo-7fd7648898-6hh48 1/1 Running 0 6s

To expose our pod as a service, type the following:

$ k expose deployment echo --type=NodePort --port=8080 service/echo exposed

Exposing the service as type NodePort means that it is exposed to the host on some port. But it is not the 8080 port we ran the pod on. Ports get mapped in the cluster. To access the service, we need the cluster IP and exposed port:

``` $ mk ip 172.26.246.89
$ k get service echo -o jsonpath='{.spec.ports[0].nodePort}' 32649 ```

Now we can access the echo service, which returns a lot of information :

``` n$ curl http://172.26.246.89:32649/hi
Hostname: echo-7fd7648898-6hh48
Pod Information: -no pod information available-
Server values: server_version=nginx: 1.14.2 - lua: 10015
Request Information: clientaddress=172.17.0.1 method=GET real path=/hi query= requestversion=1.1 requestscheme=http requesturi=http://172.26.246.89:8080/hi
Request Headers: accept=/ host=172.26.246.89:32649 user-agent=curl/7.79.1
Request Body: -no body in request- ```

Congratulations! You just created a local Kubernetes cluster, deployed a service and exposed it to the world.

Examining the cluster with the dashboard

Kubernetes has a very nice web interface, which is deployed, of course, as a service in a pod. The dashboard is well designed and provides a high-level overview of your cluster as well as drilling down into individual resources, viewing logs, editing resource files, and more. It is the perfect weapon when you want to check out your cluster manually and don't local tools like KUI ot Lens. Minikube provides it as an addon.Let's enable it:

``` $ mk addons enable dashboard ▪ Using image kubernetesui/dashboard:v2.3.1 ▪ Using image kubernetesui/metrics-scraper:v1.0.7 💡 Some dashboard features require the metrics-server addon. To enable all features please run:
    minikube addons enable metrics-server
🌟 The 'dashboard' addon is enabled ```

To launch it, type:

$ mk dashboard 🤔 Verifying dashboard health ... 🚀 Launching proxy ... 🤔 Verifying proxy health ... 🎉 Opening http://127.0.0.1:63200/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/ in your default browser...

Minikube will open a browser window with the dashboard UI.Here is the workloads view, which displays deployments, replica sets, and pods.

Figure 2.5 - dashboard

It can also display daemon sets, stateful sets, and jobs, but we don't have any in this cluster.In this section, we created a local single-node Kubernetes cluster on Windows, explored it a little bit using kubectl, deployed a service, and played with the web UI. In the next section, we'll move to a multi-node cluster.

Creating a multi-node cluster with KinD

In this section, we'll create a multi-node cluster using KinD. We will also repeat the deployment of the echo server we deployed on Minikube and observe the differences. Spoiler alert - everything will be faster and easier!

Quick introduction to KinD

KinD stands for Kubernetes in Docker. It is a tool for creating ephemeral clusters (no persistent storage). It was built primarily for running the Kubernetes conformance tests. It supports Kubernetes 1.11+. Under the covers it uses kubeadm to bootstrap Docker containers as nodes in the cluster. KinD is a combination of a library and a CLI. You can use the library in your code for testing or other purposes. KinD can create highly-available clusters with multiple control plane nodes. Finally, KinD is a CNCF conformant Kubernetes installer. It had better be if it's used for the conformance tests of Kubernetes itself :-).KinD is super fast to start, but it has some limitations too: - No persistent storage - No support for alternative runtimes yet, only DockerLet's install KinD and get going.

Installing KinD

You must have Docker installed as KinD is literally running as a Docker container. If you have Go installed you can install the KinD CLI via:

go install sigs.k8s.io/kind@v0.14.0

Otherwise, on macOS type:

brew install kind

On Windows type:

choco install kind

Dealing with Docker Contexts

You may have multiple docker engines on your system and the docker context determines, which one is used. You may get an error like:

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

In this case check your docker contexts:

$ docker context ls NAME DESCRIPTION DOCKER ENDPOINT KUBERNETES ENDPOINT ORCHESTRATOR colima colima unix:///Users/gigi.sayfan/.colima/docker.sock default * Current DOCKER_HOST based configuration unix:///var/run/docker.sock https://127.0.0.1:6443 (default) swarm rancher-desktop Rancher Desktop moby context unix:///Users/gigi.sayfan/.rd/docker.sock https://127.0.0.1:6443 (default)

The context marked with * is the current context. If you use Rancher Desktop then you choose set the context to rancher-desktop:

$ docker context use rancher-desktop

Creating the cluster with KinD

Creating a cluster is super-easy.

``` $ kind create cluster Creating cluster "kind" ... ✓ Ensuring node image (kindest/node:v1.23.4) 🖼 ✓ Preparing nodes 📦 ✓ Writing configuration 📜 ✓ Starting control-plane 🕹️ ✓ Installing CNI 🔌 ✓ Installing StorageClass 💾 Set kubectl context to "kind-kind" You can now use your cluster with:
kubectl cluster-info --context kind-kind
Thanks for using kind! 😊 ```

It takes less than 30 seconds to create a single node cluster.Now, we can access the cluster using kubectl

``` $ k config current-context kind-kind
$ k cluster-info Kubernetes control plane is running at https://127.0.0.1:51561 CoreDNS is running at https://127.0.0.1:51561/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. ```

Kind adds its kube context to the default ~/.kube/config file by default. When creating a lot of temporary clusters it is sometimes better to store the kind contexts in separate files and avoid cluttering ~/.kube/config. This is easily done by passing --kubeconfig flag with a file path. So, kind creates a single node cluster by default:

$ k get no NAME STATUS ROLES AGE VERSION kind-control-plane Ready control-plane,master 4m v1.23.4

Let's delete it and create a multi-node cluster:

$ kind delete cluster Deleting cluster "kind" ...

To create a multi-node cluster we need to provide a configuration file with the specification of our nodes. Here is a configuration file that will create a cluster called multi-node-cluster with one control-plane node and 2 worker nodes:

kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 name: multi-node-cluster nodes: - role: control-plane - role: worker - role: worker

Let's save the configuration file as kind-multi-node-config.yaml and create the cluster stroing the kubeconfig into its own file $TMPDIR/kind-multi-node-config

``` $ kind create cluster --config kind-multi-node-config.yaml --kubeconfig $TMPDIR/kind-multi-node-config Creating cluster "multi-node-cluster" ... ✓ Ensuring node image (kindest/node:v1.23.4) 🖼 ✓ Preparing nodes 📦 📦 📦 ✓ Writing configuration 📜 ✓ Starting control-plane 🕹️ ✓ Installing CNI 🔌 ✓ Installing StorageClass 💾 ✓ Joining worker nodes 🚜 Set kubectl context to "kind-multi-node-cluster" You can now use your cluster with:
kubectl cluster-info --context kind-multi-node-cluster --kubeconfig /var/folders/qv/7l781jhs6j19gw3b89f4fcz40000gq/T//kind-multi-node-config
Have a nice day! 👋 ```

Yeah, it works! and we got a local 3 nodes cluster in less than a minute:

$ k get nodes --kubeconfig $TMPDIR/kind-multi-node-config NAME STATUS ROLES AGE VERSION multi-node-cluster-control-plane Ready control-plane,master 2m17s v1.23.4 multi-node-cluster-worker Ready <none> 100s v1.23.4 multi-node-cluster-worker2 Ready <none> 100s v1.23.4

KinD is also kind enough (see what I did there) to let us create HA (highly available clusters) with multiple control plane nodes for redundancy. If you want a highly available cluster with three control plane nodes and two worker nodes your cluster config file will be very similar:

kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 name: ha-multi-node-cluster nodes: - role: control-plane - role: control-plane - role: control-plane - role: worker - role: worker

Let's save the configuration file as kind-ha-multi-node-config.yaml and create a new HA cluster

``` $ kind create cluster --config kind-ha-multi-node-config.yaml --kubeconfig $TMPDIR/kind-ha-multi-node-config Creating cluster "ha-multi-node-cluster" ... ✓ Ensuring node image (kindest/node:v1.23.4) 🖼 ✓ Preparing nodes 📦 📦 📦 📦 📦 ✓ Configuring the external load balancer ⚖️ ✓ Writing configuration 📜 ✓ Starting control-plane 🕹️ ✓ Installing CNI 🔌 ✓ Installing StorageClass 💾 ✓ Joining more control-plane nodes 🎮 ✓ Joining worker nodes 🚜 Set kubectl context to "kind-ha-multi-node-cluster" You can now use your cluster with:
kubectl cluster-info --context kind-ha-multi-node-cluster --kubeconfig /var/folders/qv/7l781jhs6j19gw3b89f4fcz40000gq/T//kind-ha-multi-node-config
Not sure what to do next? 😅 Check out https://kind.sigs.k8s.io/docs/user/quick-start/
```

Hmmm... there is something new here. Now kind creates an external load balancer as well as joining more control-plane nodes before joining the worker nodes. The load balancer is necessary to distribute requests across all the control plane nodes.Note that the external load balancer doesn't show as a node using kubectl

$ k get nodes --kubeconfig $TMPDIR/kind-ha-multi-node-config NAME STATUS ROLES AGE VERSION ha-multi-node-cluster-control-plane Ready control-plane,master 3m31s v1.23.4 ha-multi-node-cluster-control-plane2 Ready control-plane,master 3m19s v1.23.4 ha-multi-node-cluster-control-plane3 Ready control-plane,master 2m22s v1.23.4 ha-multi-node-cluster-worker Ready <none> 2m4s v1.23.4 ha-multi-node-cluster-worker2 Ready <none> 2m5s v1.23.4

But, kind has its own get nodes command, where you can see the load balancer:

$ kind get nodes --name ha-multi-node-cluster ha-multi-node-cluster-control-plane2 ha-multi-node-cluster-external-load-balancer ha-multi-node-cluster-control-plane ha-multi-node-cluster-control-plane3 ha-multi-node-cluster-worker ha-multi-node-cluster-worker2

Doing work with KinD

Let's deploy our echo service on the kind cluster. It starts the same:

``` $ k create deployment echo --image=g1g1/echo-server:0.1 --kubeconfig $TMPDIR/kind-ha-multi-node-config deployment.apps/echo created
$ k expose deployment echo --type=NodePort --port=7070 --kubeconfig $TMPDIR/kind-ha-multi-node-config service/echo exposed ```

Checking our services we can see the echo service front and center:

$ k get svc echo --kubeconfig $TMPDIR/kind-ha-multi-node-config NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE echo NodePort 10.96.52.33 <none> 7070:31953/TCP 10s

But, there is no external IP to the service. With minikube we got the IP of the minikube node itself via $(minikube ip) and we could use it in combination with the node port to access the service. That is not an option with kind clusters. Let's see how to use a proxy to access the echo service.

Accessing Kubernetes services locally though a proxy

We will go into a lot of detail about networking, services and how to expose them outside the cluster later in the book.Here, I just show how to get it done and keep you in suspense for now. First, we need to run the kubectl proxy command that exposes the API server, pods and services on localhost:

$ k proxy --kubeconfig $TMPDIR/kind-ha-multi-node-config & [1] 32479 Starting to serve on 127.0.0.1:8001

Then, we can access the echo service though a specially crafted proxy URL that includes the exposed port (8080) and NOT the node port.I use httpie here. You can use curl too. To install httpie follow the instructions here: https://httpie.org/doc#installation

``` $ http http://localhost:8001/api/v1/namespaces/default/services/echo:7070/proxy/yeah-it-works HTTP/1.1 200 OK Audit-Id: 294cf10b-0d60-467d-8a51-4414834fc173 Cache-Control: no-cache, private Content-Length: 13 Content-Type: text/plain; charset=utf-8 Date: Mon, 23 May 2022 21:54:01 GMT
yeah-it-works ```

We will deep dive into exactly what's going on in a future chapter. Let's check out my favorite local cluster solution - k3d.

Creating a multi-node cluster with k3d

In this section, we'll create a multi-node cluster using k3d from Rancher. We will not repeat the deployment of the echo server because it's identical to the KinD cluster including accessing it though a proxy. Spoiler alert - it is even faster and more user-friendly than KinD!

Quick introduction to k3s and k3d

Rancher created k3s, which is a lightweight Kubernetes distribution. Rancher says that k3s is 5 less than k8s if that makes any sense. The basic idea is to remove features and capabilities that most people don't need such as:

Non-default features

Legacy features

Alpha features

In-tree storage drivers

In-tree cloud providers

k3s removed Docker completely and uses containerd instead. You can still bring Docker back if you depend on it. Another major change is that k3s stores its state in a sqlite DB instead of etcd. For networking and DNS k3s uses Flannel and CoreDNS.k3s also added a simplified installer that takes care of SSL and certificate provisioning.The end result is astonishing - a single binary (less than 40MB) that needs only 512MB of memory. Unlike Minikube and KinD, k3s is actually designed for production. The primary use case is for edge computing, IoT and CI systems. It is optimized for ARM devices.OK. That's k3s, but what's k3d? k3d takes all the goodness that is k3s and packages it in Docker (similar to KinD) and adds a friendly CLI to manage it.

Installing k3d

Installing k3d on the MacOs is as simple as:

brew install k3d

And on Windows is just:

choco install -y k3d

On Windows, optionally add this alias to your WSL .bashrc file:

alias k3d='k3d.exe'

Let's see what we have:

$ k3d version k3d version v5.4.1 k3s version v1.22.7-k3s1 (default)

Creating the cluster with k3d

Are you ready to be amazed? Creating a single node cluster with k3d takes less than 20 seconds!

``` $ time k3d cluster create INFO[0000] Prep: Network INFO[0000] Created network 'k3d-k3s-default' INFO[0000] Created image volume k3d-k3s-default-images INFO[0000] Starting new tools node... INFO[0000] Starting Node 'k3d-k3s-default-tools' INFO[0001] Creating node 'k3d-k3s-default-server-0' INFO[0001] Creating LoadBalancer 'k3d-k3s-default-serverlb' INFO[0002] Using the k3d-tools node to gather environment information INFO[0002] HostIP: using network gateway 172.19.0.1 address INFO[0002] Starting cluster 'k3s-default' INFO[0002] Starting servers... INFO[0002] Starting Node 'k3d-k3s-default-server-0' INFO[0008] All agents already running. INFO[0008] Starting helpers... INFO[0008] Starting Node 'k3d-k3s-default-serverlb' INFO[0015] Injecting records for hostAliases (incl. host.k3d.internal) and for 2 network members into CoreDNS configmap... INFO[0017] Cluster 'k3s-default' created successfully! INFO[0018] You can now use it like this: kubectl cluster-info
real 0m18.154s user 0m0.005s sys 0m0.000s ```

Without load balancer it takes less than 8 seconds!What about multi-node cluster? We saw that KinD was much slower, especially when creating a HA cluster with multiple control-plane nodes and external load balancer.Let's delete the single node cluster first:

$ k3d cluster delete INFO[0000] Deleting cluster 'k3s-default' INFO[0000] Deleting cluster network 'k3d-k3s-default' INFO[0000] Deleting 2 attached volumes... WARN[0000] Failed to delete volume 'k3d-k3s-default-images' of cluster 'k3s-default': failed to find volume 'k3d-k3s-default-images': Error: No such volume: k3d-k3s-default-images -> Try to delete it manually INFO[0000] Removing cluster details from default kubeconfig... INFO[0000] Removing standalone kubeconfig file (if there is one)... INFO[0000] Successfully deleted cluster k3s-default!

Now, let's create a cluster with 3 worker nodes. That takes a little over 30 seconds

``` $ time k3d cluster create --agents 3 INFO[0000] Prep: Network INFO[0000] Created network 'k3d-k3s-default' INFO[0000] Created image volume k3d-k3s-default-images INFO[0000] Starting new tools node... INFO[0000] Starting Node 'k3d-k3s-default-tools' INFO[0001] Creating node 'k3d-k3s-default-server-0' INFO[0001] Creating node 'k3d-k3s-default-agent-0' INFO[0002] Creating node 'k3d-k3s-default-agent-1' INFO[0002] Creating node 'k3d-k3s-default-agent-2' INFO[0002] Creating LoadBalancer 'k3d-k3s-default-serverlb' INFO[0002] Using the k3d-tools node to gather environment information INFO[0002] HostIP: using network gateway 172.22.0.1 address INFO[0002] Starting cluster 'k3s-default' INFO[0002] Starting servers... INFO[0002] Starting Node 'k3d-k3s-default-server-0' INFO[0008] Starting agents... INFO[0008] Starting Node 'k3d-k3s-default-agent-0' INFO[0008] Starting Node 'k3d-k3s-default-agent-2' INFO[0008] Starting Node 'k3d-k3s-default-agent-1' INFO[0018] Starting helpers... INFO[0019] Starting Node 'k3d-k3s-default-serverlb' INFO[0029] Injecting records for hostAliases (incl. host.k3d.internal) and for 5 network members into CoreDNS configmap... INFO[0032] Cluster 'k3s-default' created successfully! INFO[0032] You can now use it like this: kubectl cluster-info
real 0m32.512s user 0m0.005s sys 0m0.000s ```

Let's verify the cluster works as expected:

``` $ k cluster-info Kubernetes control plane is running at https://0.0.0.0:60490 CoreDNS is running at https://0.0.0.0:60490/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy Metrics-server is running at https://0.0.0.0:60490/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. ```

Here are the nodes. Note that there is just one control plane node called k3d-k3s-default-server-0:

$ k get nodes NAME STATUS ROLES AGE VERSION k3d-k3s-default-server-0 Ready control-plane,master 5m33s v1.22.7+k3s1 k3d-k3s-default-agent-0 Ready <none> 5m30s v1.22.7+k3s1 k3d-k3s-default-agent-2 Ready <none> 5m30s v1.22.7+k3s1 k3d-k3s-default-agent-1 Ready <none> 5m29s v1.22.7+k3s1

You can stop and start clusters, create multiple clusters and list existing clusters using the k3d CLI. Here are all the commands. Feel free to explore further:

``` $ k3d Usage: k3d [flags] k3d [command]
Available Commands: cluster Manage cluster(s) completion Generate completion scripts for [bash, zsh, fish, powershell | psh] config Work with config file(s) help Help about any command image Handle container images. kubeconfig Manage kubeconfig(s) node Manage node(s) registry Manage registry/registries version Show k3d and default k3s version
Flags: -h, --help help for k3d --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) --version Show k3d and default k3s version
Use "k3d [command] --help" for more information about a command. ```

You can repeat the steps for deploying, exposing and accessing the echo service on your own. It works just like KinD.OK. We created clusters using minikube, KinD and k3d. Let's compare them, so you can decide, which one works for you.

Comparing Minikube, KinD and k3d

Minikube is an official local Kubernetes release. It's very mature and very full-featured. That said, it requires a VM and is both slow to install and to start. It also can get into trouble with networking at arbitrary times and sometimes the only remedy is deleting the cluster and rebooting. Also, minikube supports a single node only. I suggest to use Minikube only if it supports some feature that you need that is not available in either KinD or k3d. See https://minikube.sigs.k8s.io/ for more info.KinD is much faster than Minikube and is used for Kubernetes conformance tests, so by definition it is a conformant Kubernetes distribution. It is the only local cluster solution that provides HA cluster with multiple control-plane nodes. It is also designed to be used as a library, which I don't find as a big attraction because it is very easy to automate CLIs from code. The main downside of KinD for local development is that it is ephemeral. I recommend to use KinD if you contribute to Kubernetes itself and want to test against it. See https://kind.sigs.k8s.io/k3d is the clear winner for me. Lightning fast, supports multiple clusters and multiple worker nodes per cluster. Easy to stop and start clusters without losing state. See https://k3d.io/

Honorable Mention - Rancher-Desktop Kubernetes Cluster

I use Rancher-Desktop as my docker engine provider, but it also comes with a built-in Kubernetes cluster. You don't get to customize it and you can't have multiple clusters or even multiple nodes in the same cluster. But, if all you need is a local single node Kubernetes cluster to play with then the rancher-desktop clusters is there for you. To use this cluster type:

$ kubectl config use-context rancher-desktop Switched to context "rancher-desktop".

You can decide how many resources you allocate to its node, which is important if you try to deploy a lot of workloads on it, because you get just the one node.

Figure 2.6 - rancher

Alright. Let's take a look at the cloud.

Creating clusters in the cloud (GCP, AWS, Azure, Digital Ocean)

Creating clusters locally is fun. It's also important during development and when trying to troubleshoot problems locally. But, in the end, Kubernetes is designed for cloud-native applications (applications that run in the cloud). Kubernetes doesn't want to be aware of individual cloud environments because that doesn't scale. Instead, Kubernetes has the concept of a cloud-provider interface. Every cloud provider can implement this interface and then host Kubernetes.

The cloud-provider interface

The cloud-provider interface is a collection of Go data types and interfaces. It is defined in a file called cloud.go, available at: https://github.com/kubernetes/cloud-provider/blob/master/cloud.go Here is the main interface:

type Interface interface { Initialize(clientBuilder ControllerClientBuilder, stop <-chan struct{}) LoadBalancer() (LoadBalancer, bool) Instances() (Instances, bool) InstancesV2() (InstancesV2, bool) Zones() (Zones, bool) Clusters() (Clusters, bool) Routes() (Routes, bool) ProviderName() string HasClusterID() bool }

This is very clear. Kubernetes operates in terms of instances, zones, clusters, and routes, and also requires access to a load balancer and provider name. The main interface is primarily a gateway. Most methods return yet other interfaces.For example, the Clusters interface is very simple:

type Clusters interface { ListClusters(ctx context.Context) ([]string, error) Master(ctx context.Context, clusterName string) (string, error) }

The ListClusters() method returns cluster names. The Master() method returns the IP address or DNS name of the control plane of the cluster.The other interfaces are not much more complicated. The entire file is 313 lines long (at the time of writing) including lots of comments. The take-home point is that it is not too complicated to implement a Kubernetes provider if your cloud utilizes those basic concepts.

Creating Kubernetes Clusters in the Cloud

Before we look at the cloud providers and their support for managed and non-managed Kubernetes let's consider how you should create and maintain clusters. If you commit to a single cloud provider, and you are happy with using their tooling then you are set. All cloud providers let you create and configure Kubernetes clusters using either a Web UI, a CLI or an API. However,if you prefer a more general approach and want to utilize GitOps to manage yuor clusters, You should look into Infrastructure as Code solutions such as Terraform and Pulumi.If you prefer to roll-out non-managed Kubernetes clusters in the cloud then Kops is a strong candidate. See: https://kops.sigs.k8s.ioLater in the book, we will discuss in detail the topic of multi-cluster provisioning and management. There are many technologies, open source projects and commercial products in this space.For now, let's look at the various cloud providers.

GCP

The Google Cloud Platform (GCP) supports Kubernetes out of the box. The so-called Google Kubernetes Engine (GKE) is a container management solution built on Kubernetes. You don't need to install Kubernetes on GCP, and you can use the Google cloud API to create Kubernetes clusters and provision them. The fact that Kubernetes is a built-in part of the GCP means it will always be well integrated and well tested, and you don't have to worry about changes in the underlying platform breaking the cloud-provider interface.If you prefer to manage Kubernetes yourself then you can just deploy it directly on GCP instances (or use Kops alpha support for GCP), but I would generally advise against it as GKE does a lot of work for you and it's integrated deeply with GCP compute, networking and core services. All in all, if you plan to base your system on Kubernetes and you don't have any existing code on other cloud platforms, then GCP is a solid choice. It leads the pack in terms of maturity, polish, depth of integration to GCP services and is usually the first to update to newer versions of Kubernetes.I spent a lot of time with Kubernetes on GKE, managing tens of clusters, upgrading them and deploying workloads. GKE is production-grade Kubernetes for sure.

GKE Autopilot

GKE also has the Autopilot project, which takes care of managing worker nodes and node pools for you, so you focus on deploying and configuring workloads.See: https://cloud.google.com/kubernetes-engine/docs/concepts/autopilot-overview

AWS

AWS has its own container management service called ECS, that is not based on Kubernetes. It also has a managed Kubernetes service called EKS. You can run Kubernetes yourself on AWS EC2 instances. Let's talk about how to roll your own Kubernetes first and then we'll discuss EKS

Kubernetes on EC2

AWS was a supported cloud provider from the get go. There is a lot of documentation on how to set it up. While you could provision some EC2 instances yourself and use kubeadm to create a cluster, I recommend using the kops (Kubernetes Operations) project mentioned earlier. Kops initially supported only AWS.It supports the following features:

Automated Kubernetes cluster CRUD for the cloud (AWS)

Highly Available (HA) Kubernetes clusters

Uses a state-sync model for dry-run and automatic idempotency

Custom support for kubectl add-ons

Kops can generate Terraform configuration

Based on a simple meta-model defined in a directory tree

Easy command-line syntax

Community support

To create a cluster, you need to do some IAM and DNS configuration, set up a S3 bucket to store the cluster configuration, and then run a single command:

kops create cluster \ --name=${NAME} \ --cloud=aws \ --zones=us-west-2a \ --discovery-store=s3://prefix-example-com-oidc-store/${NAME}/discovery

The complete instructions are here: https://kops.sigs.k8s.io/getting_started/aws/At the end of 2017 AWS joined the CNCF and made two big announcements regarding Kubernetes: Its own Kubernetes-based container orchestration solution (EKS) and a container-on-demand solution (Fargate).

AWS EKS

EKS (Amazon Elastic Container Service for Kubernetes) is a fully managed and highly available Kubernetes solution. It has three control plane nodes running in three AZs. EKS also takes care of upgrades and patching. The great thing about EKS is that it runs a stock Kubernetes. This means you can use all the standard plugins and tools developed by the community. It also opens the door to convenient cluster federation with other cloud providers and/or your own on-premise Kubernetes clusters.EKS provides deep integration with AWS infrastructure like IAM authentication is integrated with Kubernetes RBAC (Role-based access control)You can also use PrivateLink if you want to access your Kubernetes masters directly from your own Amazon VPC. With PrivateLink, your Kubernetes control plane and the Amazon EKS service endpoint appear as an elastic network interface with private IP addresses in your Amazon VPC.Another important piece of the puzzle is a special CNI plugin that lets your Kubernetes components talk to each other using AWS networking.EKS keeps getting better and Amazon demonstrated that it is committed to keeping it up to date and improving it. If you are an AWS shop and getting into Kubernetes I recommend to start with EKS as opposed to building your own cluster.The eksctl tool is a great CLI for creating and managing EKS clusters and node groups for testing and development. I successfully created, deleted and added nodes to several Kubernetes clusters on AWS using eksctl. Check out https://eksctl.io/.

Fargate

Fargate lets you run containers directly without worrying about provisioning hardware. It eliminates a huge part of the operational complexity at the cost of losing some control. When using Fargate you package your application into a container, specify CPU and memory requirements, define networking and IAM policies and you're off to the races. Fargate can run on top of ECS and EKS. It is a very interesting member in the serverless camp although it's not specific to Kubernetes like GKE's Autopilot.

Azure

Azure used to have its own container management service based on Mesos-based DC/OS or Docker Swarm to manage your containers. But you can also use Kubernetes, of course. You could also provision the cluster yourself (for example, using Azure's desired state configuration) then create the Kubernetes cluster using kubeadm. Kops has alpha support for Azure and the Kubespray project is a good option too.However, in the second half of 2017 Azure jumped on the Kubernetes bandwagon too and introduced AKS – Azure Container Service. It is similar to Amazon EKS, although it's a little further ahead in its implementation.AKS provides a Web UI, CLI and REST API to manage your Kubernetes clusters. Once, an AKS cluster is configured you can use kubectl and any other Kubernetes tooling directly.Here are some of the benefits of using AKS:

Automated Kubernetes version upgrades and patching

Easy cluster scaling

Self-healing hosted control plane (masters)

Cost savings - pay only for running agent pool nodes

AKS also offers integration with ACI (Azure Container Instances), which is similar to AWS Fargate and GKE AutoPilot. This means that not only the control plane of your Kubernetes cluster is managed, but also the worker nodes.

Digital Ocean

Digital Ocean is not a cloud provider on the order the big three (GCP, AWS, Azure), but it does provide a managed Kubernetes solution and it has data centers across the world (US, Canada, Europe, Asia). It is also much cheaper compared to the alternatives, and cost is a major decision factor when choosing a cloud provider. With digital Ocean the control plane doesn't cost anything. Besides lower prices Digital Ocean's claim to fame is simplicity.DOKS (Digital Ocean Kubernetes Service) gives you a managed Kubernetes control plane (can be highly available) and integration with Digital Ocean's droplets (for nodes and node pools), load balancers and block storage volumes. This covers all the basic needs. Your clusters are of course CNCF conformant.Digital Ocean will take care of system upgrades, security patches and the installed packages on the control plane as well as the worker nodes.

Other cloud providers

GCP, AWS and Azure are leading the pack, but there are quite a few other companies that offer managed Kubernetes services. In general, I recommend using these providers if you already have significant business connections or integrations.

Once upon a time in China

If you operate in China with its special constraints and limitations you should probably use a Chinese cloud platform. There three big ones: Alibaba, Tencent and Huawei, The Chinese Alibaba cloud is an up and comer on the cloud platform scene. It mimics AWS pretty closely, although its English documentation leaves a lot to be desired. I deployed some production application on Ali cloud, but not Kubernetes clusters. The Alibaba cloud supports Kubernetes in several ways via its ACK (Alibaba container service for Kubernetes): - Run your own dedicated Kubernetes cluster (you must create 3 master nodes and upgrade and maintain them) - Use the managed Kubernetes cluster (you're just responsible for the worker nodes) - Use the serverless Kubernetes cluster via ECI (Elastic container instances), which is similar to Fargate and ACI.ACK is a CNCF certified Kubernetes distribution. If you need to deploy cloud-native applications in China then ACK looks like a solid option.See https://www.alibabacloud.com/product/kubernetesTencent is another large Chinese company with it own cloud platform and Kubernetes support. TKE (Tencent Kubernetes engine) seems less mature than ACK. See: https://intl.cloud.tencent.com/products/tkeFinally, the Huawei cloud platform offers CCE (Cloud Container Engine), which is built on Kubernetes. It supports VMs, bare metal and GPU accelerated instances. See: https://www.huaweicloud.com/intl/en-us/product/cce.html

IBM Kubernetes Service

IBM is investing heavily in Kubernetes. It acquired RedHat at the end of 2018. RedHat was of course a major player in the Kubernetes world, building its OpenShift Kubernetes-based platform and contributing RBAC to Kubernetes. IBM has its own cloud platform and it offers a managed Kubernetes cluster. You can try it out for free with $200 credit and there is also a free tier.IBM is also involved in the development of Istio and Knative, so you can expect IKS to have deep integration with those technologies.IKS ofers integration with a lot of IBM services.See: https://www.ibm.com/cloud/kubernetes-service

Oracle Container Service

Oracle also has a cloud platform and of course it offers a managed Kubernetes service too, with high availability, bare metal instances and multi-AZ support.OKE supports ARM and GPU instances and also offers a feww control planeSee: https://www.oracle.com/cloud/cloud-native/container-engine-kubernetes/In this section, we covered the cloud-provider interface and looked at the recommended ways to create Kubernetes clusters on various cloud providers. The scene is still young and the tools evolve quickly. I believe convergence will happen soon. Kubeadm has matured and is the underlying foundation of many other tools to bootstrap and create Kubernetes clusters on and off the cloud. Let's consider now what it takes to create bare-metal clusters where you have to provision the hardware and low-level networking and storage too.

Creating a bare-metal cluster from scratch

In the previous section, we looked at running Kubernetes on cloud providers. This is the dominant deployment story for Kubernetes. But there are strong use cases for running Kubernetes on bare metal, such Kubernetes on the edge. I don't focus here on hosted versus on-premise. This is yet another dimension. If you already manage a lot of servers on-premise, you are in the best position to decide.

Use cases for bare-metal

Bare-metal clusters are a bear, especially if you manage them yourself. There are companies that provide commercial support for bare-metal Kubernetes clusters, such as Platform 9, but the offerings are not mature yet. A solid open-source option is Kubespray, which can deploy industrial-strength Kubernetes clusters on bare metal, AWS, GCE, Azure and OpenStack.Here are some use cases where it makes sense:

Price : If you already manage large-scale bare clusters, it may be much cheaper to run Kubernetes clusters on your physical infrastructure

Low network latency : If you must have low latency between your nodes, then the VM overhead might be too much

Regulatory requirements : If you must comply with regulations, you may not be allowed to use cloud providers

You want total control over hardware : Cloud providers give you many options, but you may have special needs

When should you consider creating a bare-metal cluster?

The complexities of creating a cluster from scratch are significant. A Kubernetes cluster is not a trivial beast. There is a lot of documentation on the Web on how to set up bare-metal clusters, but as the whole ecosystem moves forward, many of these guides get out of date quickly.You should consider going down this route if you have the operational capability to troubleshoot problems at every level of the stack. Most of the problems will probably be networking-related, but filesystems and storage drivers can bite you too, as well as general incompatibilities and version mismatches between components such as Kubernetes itself, Docker (or other runtimes, if you use them), images, your OS, your OS kernel, and the various add-ons and tools you use. If you opt for using VMs on top of bare metal then you add another layer of complexity.

Understanding the process

There is a lot to do. Here is a list of some of the concerns you'll have to address:

Implementing your own cloud-provider interface or sidestepping it

Choosing a networking model and how to implement it (CNI plugin, direct compile)

Whether or not to use network policies

Select images for system components

Security model and SSL certificates

Admin credentials

Templates for components such as API Server, replication controller, and scheduler

Cluster services: DNS, logging, monitoring, and GUI

I recommend the following guide from the Kubernetes site to get a deeper understanding of what it takes to create a HA cluster from scratch using kubeadm: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/

Using the Cluster API for managing bare metal clusters

The Cluster API (AKA CAPI) is a Kubernetes sub-project for managing Kubernetes clusters at scale. It uses kubeadm for provisioning. It can provision and manage Kubernetes clusters in any environment using providers. At work, we use it to manage multiple clusters in the cloud. But, it has multiple providers for bare metal clusters:

MAAS

Equinix metal

Metal3

Cidero

See: https://cluster-api.sigs.k8s.io

Using virtual private cloud infrastructure

If your use case falls under the bare-metal use cases but you don't have the necessary skilled manpower or the inclination to deal with the infrastructure challenges of bare metal, you have the option to use a private cloud such as OpenStack. If you want to aim a little higher in the abstraction ladder, then Mirantis offers a cloud platform built on top of OpenStack and Kubernetes.Let's review a few more tools for building Kubernetes clusters on bare metal. Some of these tools support OpenStack as well.

Building your own cluster with Kubespray

Kubespray is a project for deploying production-ready highly available Kubernetes clusters. It uses Ansible and can deploy Kubernetes on a large number of targets such as:

AWS

GCE

Azure

OpenStack

vSphere

Equinix metal

Oracle Cloud Infrastructure (Experimental)

And also to plain bare metal.It is highly customizable and support multiple operating systems for the nodes, multiple CNI plugins for networking and multiple container runtimes. If you want to test it locally it can deploy to a multi-node vagrant setup too. If you're an Ansible fan Kubespray may be a great choice for you.See: https://kubespray.io

Building your cluster with Rancher RKE

RKE (Rancher Kubernetes Engine) is a friendly Kubernetes installer that can install Kubernetes on on bare-metal as well as virtualized servers. RKE aims to address the complexity of installing Kubernetes. It is open source and has great documentation. Check it out here: http://rancher.com/docs/rke/v0.1.x/en/

Running managed Kubernetes on bare metal or VMs

The cloud providers didn't want to confine themsleves to their own cloud only. They all offer multi-cloud and hybrid solution where you can control Kubernetes cluster on multiple clouds as well as use their managed control plane on VMs anywhere.

GKE Anthos

Anthos is a managed platform for all your application deployments, both traditional as well as cloud native. It enables you to build and manage global fleets and establish operational consistency across them.

EKS Anywhere

Amazon EKS Anywhere is a new deployment option for Amazon EKS that allows you to create and operate Kubernetes clusters on your infrastructure, supported by AWS. You can now run Amazon EKS Anywhere on on-premises infrastructure using VMware vSphere and soon for bare metal too.

AKS Arc

Azure Arc is a set of technologies that brings Azure security and cloud-native services to hybrid and multicloud environments. It enables you to secure and govern your infrastructure and apps anywhere, build cloud-native apps faster with familiar tools and services to run them on any Kubernetes platform.

Summary

In this chapter, we got into some hands-on cluster creation. We created a single-node and multi-node clusters using tools like Minikube, Kind and k3d. Then we looked at the various options to create Kubernetes clusters on cloud providers. Finally, we touched on the complexities of creating Kubernetes clusters on bare metal. The current state of affairs is very dynamic. The basic components are changing rapidly, the tooling is getting better, and there are different options for each environment. Kubeadm is now the cornerstone of most installation options, which is great for consistency and consolidation of effort. It's still not completely trivial to stand up a Kubernetes cluster on your own, but with some effort and attention to detail you can get it done quickly.I highly recommend to consider the Cluster API as the go to solution for provisioning and managing clusters in any environment - managed, private cloud, VMs and bare metal.In the next chapter, we will explore the important topics of scalability and high-availability. Once your cluster is up and running you need to make sure it stays that way even as the volume of requests increases. This requires ongoing attention and building the ability to recover from failures as well adjusting to changes in traffic

References

https://alexos.dev/2022/01/02/docker-desktop-alternatives-for-m1-mac/https://rancher.com/docs/rke/latest/en/https://github.com/kubernetes-sigs/kubesprayhttps://kubespray.io https://rancher.com/docs/rke/latest/en/https://cloud.google.com/anthos https://aws.amazon.com/eks/eks-anywhere/https://docs.microsoft.com/en-us/azure/azure-arc/kubernetes/https://thenewstack.io/provision-bare-metal-kubernetes-with-the-cluster-api/

Join our book community on Discord

https://packt.link/K8s