Ultimate Certified
Kubernetes
Administrator (CKA)
Certification Guide

Become CKA Certified with Ease by Mastering
Cluster Management and Orchestration with
Kubernetes

Rajesh Vishnupant Gheware

www.orangeava.com

Copyright © 2024 Orange Education Pvt Ltd, AVA™

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author nor Orange Education Pvt Ltd or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Orange Education Pvt Ltd has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capital. However, Orange Education Pvt Ltd cannot guarantee the accuracy of this information. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use.

First published: July 2024

Published by: Orange Education Pvt Ltd, AVA™

Address: 9, Daryaganj, Delhi, 110002, India

275 New North Road Islington Suite 1314 London,

N1 7AA, United Kingdom

ISBN: 978-81-97651-16-8

www.orangeava.com

Dedicated To

Prof. L. S. Ganesh Sir (IIT Madras)

For Being my Unending Source of Inspiration,

Your Guidance and Wisdom have Shaped
my Professional Journey

My Parents,

who I am Eternally Indebted for Shaping
my Early Years of Life

My Bundles of Joy - Shreyansi and Naina,

Your Boundless Love Fuels my Ambitions,

May you Both Always Shine Brightly

My Best Critic Aka Wife - Deepti, for Your Unwavering
Support

This Book Is Dedicated to You All,

With Heartfelt Gratitude and Love,

For Being the Pillars of my Life

About the Author

Rajesh Gheware is a seasoned professional in the IT industry, known for his expertise and contributions in the field of DevOps. With over two decades of experience, Rajesh has made a significant impact in the areas of cloud computing, containerization, and strategic IT architectures. His career has been marked by progressive roles, starting as a software engineer and evolving into a Chief Architect, a position he has held at several prestigious organizations.

Rajesh's journey in technology began with a strong foundation in engineering, having earned an M.Tech from IIT Madras. His passion for technology, combined with a keen understanding of business needs, has enabled him to lead complex IT projects and strategies successfully. Rajesh’s technical skills are diverse, including expertise in Kubernetes, Docker, AWS services, and various software frameworks and protocols.

Rajesh boasts a diverse IT career with impactful roles at UniGPS Solutions, JP Morgan Chase, and Deutsche Bank. He spearheaded IT strategies and cutting-edge solutions, particularly in cloud computing and containerization, setting industry benchmarks. A thought leader in DevOps, Rajesh actively contributes to platforms like DZone, LinkedIn, and OpenSourceForU, offering valuable insights into DevOps tools and methodologies.

Rajesh's expertise extends to mentoring. His workshops and training sessions in Kubernetes, Docker, AWS, and DevOps practices have established him as a sought-after speaker and educator with an approachable and informative style. Known for his problem-solving skills, mentorship, and contributions to DevOps best practices, Rajesh is a respected figure in the community, driven by continuous learning and knowledge sharing.

Technical Review Partner

Thinknyx® Technologies is a team of professionals with years of experience in IT technology, ranging from Software Development to the Management of IT Infrastructure, Cloud, Automation, Container Management, Web and APP Development, Security, and Professional Services. Recognized as a reputable brand, Thinknyx® Technologies provides IT consulting services, offering comprehensive Information Technology and Soft Skills’ Training. Additionally, they offer Talent Acquisition and Recruitment solutions to diverse organizations worldwide.

Mr. Yogesh Raheja, the Founder and CEO of Thinknyx® Technologies, is a certified expert in DevOps, SRE, Cloud, and Containerisation, with two decades of IT experience.

He has expertise in technologies such as Public/Private Cloud, Containers, Automation tools, Continuous Integration/Deployment/Delivery tools, Monitoring and Logging tools, and more. Mr. Raheja is passionate about sharing his technical expertise with a global audience through various forums, conferences, webinars, blogs, and LinkedIn. Moreover, he has authored multiple books, including Effective DevOps with AWS, Automation with Puppet 5, and Automation with Ansible, and has published his online courses on various platforms. Furthermore, he has reviewed multiple books, including Implementing Splunk 7, Third Edition and Splunk Operational Intelligence Cookbook, Third Edition, among others.

Acknowledgements

I am immensely grateful to everyone who supported me through the journey of writing this book. First and foremost, I extend my deepest appreciation to my youngest daughter, Naina, whose sharp insights and dedication to refining the content, greatly enhanced the clarity and precision of this guide. Her involvement has been invaluable, and I am profoundly thankful for her continuous help.

I also owe a huge debt of gratitude to all my family members for their unwavering patience and support. Their understanding and encouragement have been crucial as I navigate the various facets of life, always striving to learn more and share that knowledge through writing.

To all those who lent their expertise, provided feedback, and encouraged me along the way—thank you. Your collective wisdom and support have not only made this guide possible but have also made it better in every conceivable way.

Thank you all for being my pillars during this endeavor.

Preface

This book, Ultimate Certified Kubernetes Administrator (CKA) Certification Guide, is the product of years spent navigating the intricacies of IT architecture and witnessing the transformative impact of technology, particularly in cloud computing and containerization. The goal is to provide a comprehensive resource that empowers IT professionals. It aims to equip them for the CKA and CKS certifications while fostering a deeper understanding of Kubernetes as a vital tool for modern infrastructure management. By delving into the world of Kubernetes, readers will gain the knowledge and skills necessary to effectively manage and orchestrate containerized applications.

This guide is crafted to serve both newcomers and seasoned practitioners by breaking down complex Kubernetes concepts into understandable segments, supported by real-world applications and examples. Through this book, we share not just the technical know-how but also the strategic thinking required to harness the full potential of Kubernetes in any organizational context. Whether you're looking to certify your skills or simply aiming to enhance your technical arsenal, this guide stands as a beacon of knowledge and practical insight in the ever-evolving world of cloud-native technologies.

Chapter 1. Introduction to Kubernetes: This chapter introduces Kubernetes, detailing its evolution from Google's internal system to the leading open-source container orchestration platform. It covers the architecture, benefits, and fundamental concepts, providing a solid foundation for understanding how Kubernetes facilitates complex deployments and operations.

Chapter 2. Installing Kubernetes: This chapter delves into the practical setup of Kubernetes environments, covering essential tools, such as Oracle VirtualBox, Vagrant, and various Kubernetes installation methods, including Minikube, Microk8s, KIND, and Kubeadm. This chapter equips readers with the knowledge to successfully install and configure Kubernetes clusters on different platforms for various use cases.

Chapter 3. Workload Objects - Pod Deploy StatefulSet: This chapter delves into key Kubernetes concepts such as Pods, Deployments, and StatefulSets, detailing their configurations, uses, and management. It extensively covers container probes—startupProbe, readinessProbe, and livenessProbe—to monitor and improve the health and responsiveness of applications. It is pivotal for understanding the lifecycle of Kubernetes workloads and how to ensure their robustness and reliability. It serves as a comprehensive guide to deploying and managing stable, scalable applications in a Kubernetes environment.

Chapter 4. Service and Ingress - Exposing Apps Outside the Cluster: This chapter explores the methods and strategies for exposing Kubernetes applications to internal and external network traffic through Services and Ingress Controllers. It covers different types of services including ClusterIP, NodePort, LoadBalancer, and ExternalName, detailing their use cases and configurations. The chapter also introduces Ingress as a powerful tool for managing external access to services within a cluster. This chapter is essential for mastering the networking aspects of Kubernetes, ensuring applications are accessible and secure.

Chapter 5. Deploy and Scale - Stateless Apps: This chapter delves into the deployment and scaling of stateless applications within Kubernetes. It begins by explaining how to set application-specific runtime specifications that aid Kubernetes in effectively placing Pods on Nodes, enhancing overall performance and enabling efficient autoscaling. It highlights the role of the metrics-server, an addon component that tracks CPU and memory usage, providing crucial data for managing resources. It covers the configuration and activation of the Horizontal Pod Autoscaler (HPA), the dynamic nature of application loads and discusses the HPA's strategy to maintain cluster stability.

Chapter 6. Deployment Strategies – RollingUpdate, Recreate: This chapter explores advanced deployment strategies in Kubernetes, focusing on RollingUpdate, Recreate, Blue-Green, and Canary deployments. It begins with the straightforward Recreate strategy, which involves full replacement of old instances, potentially causing downtime. It then covers the RollingUpdate method, which allows for seamless, zero-downtime updates by incrementally replacing pods. The chapter further delves into Blue-Green deployments that facilitate testing by running two identical environments before fully switching traffic to the new version. Finally, it examines Canary deployments, which roll out changes to a small subset of users to gauge impact before a broader rollout.

Chapter 7. Data Persistence - Local and Cloud: The chapter thoroughly addresses the topic of data persistence in Kubernetes, focusing on both local and cloud storage solutions. It explores the mechanisms through which Kubernetes manages persistent data across various environments, ensuring data durability and accessibility. It details the use of Persistent Volumes (PVs) and Persistent Volume Claims (PVCs) for managing local storage resources, including their lifecycle, provisioning, and dynamic binding processes. The chapter also delves into cloud storage integrations, discussing how Kubernetes interfaces with cloud providers to leverage their storage services for stateful applications.

Chapter 8. Deploy and Scale – StatefulSet: This chapter is dedicated to the deployment and scaling of stateful applications using StatefulSets in Kubernetes. It provides a comprehensive understanding of StatefulSets, which are crucial for applications that require stable, unique network identifiers and persistent storage. It covers the creation, management, and scaling of StatefulSets, detailing how they maintain the order and uniqueness of pods during updates and scaling operations. The chapter also discusses strategies for effectively managing stateful resources, such as databases, within a Kubernetes environment, ensuring data consistency and high availability.

Chapter 9. Configure Apps for Production Deployment: The chapter focuses on configuring applications for production deployment in Kubernetes. This chapter provides detailed guidance on best practices for preparing applications for the rigors of a production environment. It covers essential topics such as setting up robust logging and monitoring, implementing efficient resource limits, and applying security configurations. It also discusses the importance of creating reliable liveness and readiness probes to ensure application health and resilience. By emphasizing automation and adherence to best practices, this chapter equips readers with the necessary tools and knowledge to configure their Kubernetes applications efficiently, ensuring smooth and stable production deployments.

Chapter 10. Cluster Database - Backup and Restore: The chapter is dedicated to the comprehensive management of the ETCD cluster database, which is critical for maintaining the state and configuration of Kubernetes clusters. It begins with an in-depth look at understanding ETCD and its role within Kubernetes, followed by detailed discussions on how to configure, manage, and secure ETCD clusters effectively. It covers key aspects of backup and disaster recovery strategies, ensuring data integrity and availability even in the face of system failures. The chapter also addresses scaling ETCD clusters to meet growing demands and optimizing performance for high availability.

Chapter 11. Cluster Upgrade – Kubeadm: This chapter is centered on upgrading Kubernetes clusters using kubeadm. It starts with an overview of the Kubernetes release cycle and versioning to set the stage for understanding the importance of timely and efficient upgrades. It then progresses into detailed preparation steps necessary for a successful upgrade, including backup strategies and the importance of reviewing release notes. The chapter concludes with insights into rollback strategies and troubleshooting common issues, along with best practices to ensure a smooth and effective upgrade process, making it an essential resource for anyone responsible for maintaining Kubernetes clusters.

Chapter 12. CoreDNS: This chapter thoroughly examines CoreDNS within Kubernetes, emphasizing its architectural components, deployment, and pivotal role in DNS resolution and service discovery. It details customization and configuration strategies, explores its integration with Kubernetes networking, and addresses performance, scalability, and security challenges. The chapter concludes by providing solutions for common operational issues, making it a vital resource for Kubernetes administrators.

Chapter 13. Networking - Pod Service and Ingress: This chapter dives into the Kubernetes networking model, explaining how containerized applications benefit from unique and efficient networking approaches. It covers intra-pod communication through shared network namespaces, inter-pod connectivity across the cluster via a flat network and CNI plugins, and the role of Kubernetes services in providing stable endpoints for load balancing and service discovery. Additionally, it details the mechanisms for external access to services, such as NodePort, LoadBalancer, and Ingress, highlighting their roles in external communication and access control.

Chapter 14. Kubernetes CNI: This chapter delves into the intricacies of Kubernetes Container Network Interface (CNI), crucial for the implementation and management of network plugins that facilitate secure and isolated container communications. It provides detailed insights into the functionality, selection, and configuration of popular CNI plugins, including Calico, Flannel, and Weave Net, illustrating how they integrate with kubelet to ensure efficient pod networking. This chapter is vital for administrators and developers seeking to understand and leverage the full capabilities of networking within Kubernetes environments.

Chapter 15. Kubernetes Security: This chapter offers an extensive examination of Kubernetes security, detailing its comprehensive security model which includes the API server as a central access point for authentication, authorization, and admission control. It explores the implementation of Pod Security Standards and the transition towards more streamlined security practices that enhance flexibility in policy enforcement, auditing, and warning. It also covers Role-Based Access Control (RBAC), network policies, securing the control plane, and the importance of audit logging. By integrating these security measures with TLS for secure communications, the chapter outlines how to significantly strengthen the security posture of Kubernetes environments, ensuring resource protection and adherence to best security practices.

Chapter 16. Troubleshooting: This chapter is dedicated to troubleshooting within Kubernetes environments, providing essential techniques for diagnosing and resolving common issues that arise in cluster management and application deployment. The chapter thoroughly covers health checks of cluster components, application failure analysis, service connectivity, and network troubleshooting, including DNS resolution and load balancing. It places significant emphasis on practical skills, such as effective use of `kubectl` commands, understanding Kubernetes events, and verifying configurations. Additionally, the chapter explores complex scenarios involving resource quota management, pod security, persistent storage configurations, and advanced deployment strategies like blue-green deployments. By blending theory with hands-on practices, this chapter equips Kubernetes administrators with the tools necessary to maintain security, performance, and resource efficiency, preparing them for real-world challenges and enhancing their proficiency for the Certified Kubernetes Administrator (CKA) exam.

Chapter 17. Kubernetes Production Essentials: This chapter addresses the critical steps needed to enhance a basic Kubernetes cluster to a production-grade environment by integrating essential third-party tools. The chapter provides detailed guidance on deploying the NGINX Ingress Controller for traffic management, Cert-Manager for TLS certificate automation, and Metrics Server for monitoring resources. It also explores the implementation of the EFK stack (Elasticsearch, Fluent-bit, Kibana) for advanced logging and integrates Prometheus and Grafana for in-depth monitoring and analytics. The importance of Application Performance Monitoring tools is highlighted to gain real-time insights into application performance. This comprehensive approach not only enhances cluster performance, security, and observability but also ensures the cluster's readiness for production deployments, making it robust and resilient for handling real-world workloads effectively.

Chapter 18. Microservices Observability: This chapter provides a detailed exploration of observability within Kubernetes environments, focusing on microservices architecture. It underscores the importance of comprehensive monitoring and management practices, incorporating logging, distributed tracing, and metrics collection as foundational elements. The chapter guides Kubernetes practitioners through best practices and tools like Elasticsearch, Fluent-bit, Kibana, APM Server, Prometheus, and Grafana, which are essential for building a robust observability framework. Through practical examples, including a case study on optimizing a Weather Application, the chapter illustrates how logs, tracing, and metrics—collectively the three pillars of observability—play critical roles in enhancing the understanding, performance, and reliability of microservices. This knowledge empowers IT professionals to improve the resilience and efficiency of their systems, ensuring scalable and performance-optimized deployments.

Chapter 19. Scalable Jenkins on Kubernetes: This chapter focuses on optimizing Jenkins for scalability within Kubernetes environments. The chapter dives into effective strategies for a scalable Jenkins setup, particularly highlighting the use of external storage solutions and the dynamic creation of pods for job execution using the Jenkins Kubernetes plugin. Detailed, step-by-step guidance on configuring this setup is provided, followed by practical demonstrations through an inline pipeline example. This approach not only enhances Jenkins' performance but also its flexibility, making it well-suited for cloud-native ecosystems. The chapter serves as a comprehensive guide for professionals looking to leverage Kubernetes to improve the efficiency and scalability of their Jenkins pipelines.

Chapter 20. GitOps using ArgoCD and GitHub: This chapter delves into GitOps, a modern methodology that revolutionizes DevOps by integrating Git with operational workflows for enhanced efficiency and control. The chapter thoroughly explores the use of GitHub and ArgoCD, a Kubernetes-native continuous delivery tool, to implement GitOps practices effectively. It demonstrates how these technologies can streamline the deployment process within Kubernetes environments, making it more transparent and automated. Through practical implementation, the chapter outlines how to establish a robust CI/CD pipeline combining Jenkins and ArgoCD, ensuring consistent deployment and synchronization of Kubernetes applications with their desired states in GitHub. This setup not only enhances operational efficiency but also aligns with the best practices of cloud-native application management, providing clear insights into integrating GitOps within modern DevOps frameworks.

Chapter 21. CKA Exam Mastery: This chapter serves as a crucial guide for mastering the Certified Kubernetes Administrator (CKA) exam. The chapter provides effective strategies and insights, emphasizing the necessity of a deep, comprehensive understanding of Kubernetes coupled with extensive, hands-on experience. It details a targeted approach to excel in the exam, focusing on the critical areas such as cluster architecture, workloads, services, networking, and troubleshooting. Highlighting the practical, hands-on nature of the exam, this chapter equips candidates with the knowledge and skills required to succeed in achieving one of the most esteemed certifications in cloud-native technologies.

Chapters 18, 19 and 20, while not directly part of the Certified Kubernetes Administrator (CKA) syllabus, offer invaluable knowledge for professionals operating in modern cloud-native environments. Chapter 18 delves into microservices observability, equipping practitioners with the tools and strategies needed for effective monitoring and management, crucial for maintaining robust and scalable services. Chapter 19 explores the integration of Jenkins within Kubernetes, focusing on scalable CI/CD practices that enhance development pipelines and operational efficiency. Chapter 20 introduces GitOps using ArgoCD and GitHub, presenting advanced deployment techniques that streamline and secure application management processes. These chapters provide essential insights and practical skills that transcend certification requirements, preparing professionals to tackle real-world challenges in Kubernetes deployments and operations, thus adding significant value to their expertise in the evolving landscape of cloud technologies.

Downloading the code
bundles and colored images

Please follow the link or scan the QR code to download the
Code Bundles and Images of the book:

https://github.com/OrangeAVA/Ultimate-Certified-Kubernetes-Administrator-CKA-Certification-Guide

The code bundles and images of the book are also hosted on
https://rebrand.ly/0a6fd4

In case there’s an update to the code, it will be updated on the existing GitHub repository.

Errata

We take immense pride in our work at Orange Education Pvt Ltd and follow best practices to ensure the accuracy of our content to provide an indulging reading experience to our subscribers. Our readers are our mirrors, and we use their inputs to reflect and improve upon human errors, if any, that may have occurred during the publishing processes involved. To let us maintain the quality and help us reach out to any readers who might be having difficulties due to any unforeseen errors, please write to us at :

errata@orangeava.com

Your support, suggestions, and feedback are highly appreciated.

DID YOU KNOW

Did you know that Orange Education Pvt Ltd offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.orangeava.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at: info@orangeava.com for more details.

At www.orangeava.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on AVA™ Books and eBooks.

PIRACY

If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at info@orangeava.com with a link to the material.

ARE YOU INTERESTED IN AUTHORING WITH US?

If there is a topic that you have expertise in, and you are interested in either writing or contributing to a book, please write to us at business@orangeava.com. We are on a journey to help developers and tech professionals to gain insights on the present technological advancements and innovations happening across the globe and build a community that believes Knowledge is best acquired by sharing and learning with others. Please reach out to us to learn what our audience demands and how you can be part of this educational reform. We also welcome ideas from tech experts and help them build learning and development content for their domains.

REVIEWS

Please leave a review. Once you have read and used this book, why not leave a review on the site that you purchased it from? Potential readers can then see and use your unbiased opinion to make purchase decisions. We at Orange Education would love to know what you think about our products, and our authors can learn from your feedback. Thank you!

For more information about Orange Education, please visit www.orangeava.com.

CHAPTER 1

Introduction to Kubernetes

Introduction

In the pre-Kubernetes era, the cost of deploying and managing software was considerably high. Moreover, the operations team faced formidable challenges with regard to replicating software within and across data centers, geographical regions, and for increased load. These were just a few of the numerous complex requirements, making deploying and managing a highly available distributed system a nightmare.

Google knew that these challenges would become bigger with time and thus developed a cluster management software called ‘Borg’. With this software, Google was running billions of containers every week on machines spread across multiple geographic locations. Nearly a decade later, Google open-sourced this software under the name of Kubernetes, and a year later, in 2015, handed it over to the Linux Foundation.

Structure

In this chapter, we will cover the following topics:

Overview

Benefits of Kubernetes

Kubernetes Architecture

Logical View

Dynamic View

Overview

Kubernetes is an open-source software used for the management of containerized applications. This includes managing the scalability of applications, facilitating automated deployment, and so on. In a nutshell, Kubernetes is a cluster management software that lets you create and deploy highly available distributed applications.

Benefits of Kubernetes

Following is the list of a few benefits that Kubernetes offers:

Service discovery and load balancing

Automated rollouts and rollbacks

Automated bin packing

Self-healing

Storage orchestration

Secrets and configuration management

You will learn and experience these benefits/advantages in greater detail in subsequent chapters. We will provide a brief explanation of the benefits mentioned here.

Service Discovery and Load Balancing

Kubernetes provides a service-level abstraction to expose your application. This service abstraction enables decoupling between callers of your application and the application itself. For example, a frontend application can call the backend application via the Service URL - a stable network URL that can be accessed within the cluster from anywhere. While the backend application replicas may be running on different nodes or replaced by new ones, the frontend application does not need to worry about which backend application replica responds. This is similar to applications accessing a database using the database URL without worrying about the underlying database instances.

Besides providing decoupling between the caller and the callee, Kubernetes Service also acts as a load balancer. For example, if there is more than one replica of a backend application, then Kubernetes will route the request to one of the available backend replicas. This routing is mostly round-robin in nature; however, it can be customized. This, in turn, helps cater to varying levels of load on the backend application.

Automated Rollouts and Rollbacks

Often in the enterprise world, many replicas of applications are deployed on dozens if not hundreds of servers. Now rolling out any new change would not only be cumbersome but also impractical in most cases. Using Kubernetes, you can roll out new changes or hotfixes with just a single line of command, and it will take care of automatically rolling out this change in the cluster, no matter how big it is.

In case of any issue during or post-rollout, you can roll back too! Again, using a single line of command, you can instruct Kubernetes, and it will roll back automatically.

This automated rollout and rollback are generally pretty quick, meaning that you can see the changes happening within a few minutes.

Automatic Bin Packing

In the pre-Kubernetes era, deployment procedures invariably contained details of servers on which the application was to be deployed. The Operations team would then deploy the application, ensuring adequate capacity was available on those hosts where the application was to be deployed.

In the Kubernetes world, you just need to specify the application’s runtime requirements, be it in RAM, CPU, or even parameters like disk type, GPU availability, and more. Kubernetes will then find the appropriate server(s) in the cluster to deploy the application onto. This is called Automatic Bin Packing.

Self-Healing

Some of the application pods may crash or hang, or even the node on which they are running may go off the network or keep crashing. This presents a challenging scenario to the Operations teams, who often end up spending a significant amount of effort, sometimes even sleepless nights, recovering from such failures to ensure the high availability of applications.

Kubernetes runs continuous checks through its controller component, identifies failures, and launches application pods on the available computer capacity (servers) to ensure the desired number of application replicas are running.

Storage Orchestration

Most enterprise applications need either temporary or permanent storage to work with. Kubernetes is designed to provide access to various kinds of storage through Volume APIs, regardless of whether storage is needed for the application’s current runtime, across multiple runtimes, or post-restart to maintain the application state. Kubernetes is also designed to work with many external storage providers such as AWS EBS, Azure, Google, Ondat, CephFS, GlusterFS, PortWorx, Cinder, and so on.

Secrets and Configuration Management

Sensitive information required by the applications, such as the API key, can be stored in the Secret object provided by Kubernetes. All the information kept in the Secret object is base64 encoded by Kubernetes. Information stored in these Secret objects can be referenced in the application via environment variables or container file systems.

Kubernetes provides a ConfigMap object to store runtime inputs, say environment variables. This gives the flexibility to deploy the application in various environments like test, pre-prod, prod, and more. Information stored in the Config object is made available to the application at runtime via environment variables or container file system. Kubernetes also ensures that if there is any change in the Config object, then the corresponding application is relaunched automatically to reflect the change in the Config for that particular application.

Kubernetes Architecture

Architecture can be best understood by looking at the system from various perspectives/views. We will first examine Kubernetes from a logical view. Then, to understand how different components interact with each other, we will use a dynamic view in the form of a sequence diagram.

Logical View

Now, let us understand the different components that make up the architecture of Kubernetes and how these components interact with each other.

Figure 1.1 shows the logical overview of the Kubernetes cluster:

Figure 1.1: Kubernetes Architecture - Logical View

Control Plane

The control plane consists of one or more servers, also called controller nodes, where the key components that are responsible for managing the Kubernetes cluster are deployed. All control plane components are designed for distributed scaling, and the number of nodes in the control plane is typically an odd number like 1, 3, 5, 7, and so on. Generally, it is 3 or more in the enterprise setups.

API Controller

This is the central component in Kubernetes architecture as all other parts/components interact with the API controller. This is the only component that interacts with the ETCD database to store and retrieve cluster information. To manage or operate on the cluster, the User (the external entity to the cluster) can send execution commands using Command Line Interface (CLI) known as ‘Kubectl’. The API Controller exposes itself over the network, allowing utilities like kubectl to send commands (JSON messages over HTTPS) to control the Kubernetes cluster.

ETCD Database

This component holds the state of the cluster, and the only component allowed to manage the state of the cluster is the API Controller. Cluster state information is stored in the ETCD database in the form of key and value pairs. Like other control plane components, the ETCD database is designed to be a highly available and scalable software component. It exposes itself over the network and on port 2379.

Scheduler

The main responsibility of the scheduler is to schedule pods on appropriate nodes. In Kubernetes, pods are the smallest deployable units. A pod may contain one or more containers. The scheduler watches for all newly created pods and assigns nodes to them so that Kubelet can launch those on the assigned nodes. While assigning nodes to the pod, the scheduler takes into account various computing needs required by the pod such as CPU, memory, disk type, GPU, affinity/anti-affinity preferences, and so on.

Controller (KCM - Kube Controller Manager)

Controller, also known as Kube Controller Manager (KCM), is a group of controller processes packaged together to reduce deployment complexity. Following are some of the controller processes:

Node controller: Monitors and responds whenever the node goes down.

Job controller: Watches for tasks represented as Jobs and launches them.

Endpoint Slice controller: Links pods to their service object.

ServiceAccount controller: Creates a service account that pods can use to communicate with the API Controller. For example, the CI/CD pipeline uses a service account to deploy the built application in the cluster.

Cloud Controller (CCM - Cloud Controller Manager)

CCM consists of controllers that interact with the cloud provider to provide features such as a cloud load balancer, manage the cloud provided with nodes, and set up routes in the cloud infrastructure.

Worker Plane

This plane consists of nodes where applications are typically deployed. Kubelet, Kube-proxy, and container runtime are also installed on all the worker plane nodes.

Kubelet

Kubelet, which runs on every node, interfaces with the container runtime that implements the Container Runtime Interface (CRI) to create and launch applications. There are many CRI implementations available, such as containerd, rocket, CRI-O, Docker, Mirantis, and so on. Kubelet also interfaces with a networking plugin that implements Container Network Interface (CNI). Many plugins, such as Weave, Calico, and Flannel, are available in the market. Kubelet continuously works with the API Controller to take appropriate action on the node, such as, launching applications, destroying applications, sending health status information, and so on.

Kube-proxy

Kube-proxy, too, runs on every node and routes the requests to one of the application instances. Kube-proxy either makes use of iptables or IP Virtual Server (IPVS) for highly performant request routing.

Dynamic View

Figure 1.2 captures the interaction among components when the user requests the creation of a pod:

Figure 1.2: POD Creation Sequence Diagram

As you can see in the diagram, when a user requests a pod creation, the API Controller (also known as API-server) responds after saving the request in the ETCD database. Controllers continually check for new changes in the cluster with the API server, whille the scheduler keeps track of all the pods that are yet to be assigned a node and assigns nodes as soon as the necessary conditions are met. Once a pod is assigned a node, the scheduler notifies the API server, which in turn saves this new information in the ETCD database. Meanwhile, the Kubelet obtains information about newly assigned pods and launches them using the container runtime on the assigned nodes. Kubelet also updates this to the API server, which then saves the latest state in the ETCD database.

Tip: You can use a control plane node to deploy your application in the test setup; hence, in Figure 1.1, the number of nodes is shown as 0..*, meaning 0 or more.

Conclusion

Kubernetes is designed for high availability and scalability. It includes many features that greatly reduce the operational burden on system admins and operations teams. It provides smooth integration with many cloud service providers, such as storage, network, and more. Moreover, it is a pluggable architecture that allows many alternate implementations to be part of it, as long as these implementations/plugins/adapters meet the interface specification. The Kubernetes framework is also extensible, allowing you to extend Kubernetes for your custom requirements.

In the next chapter, you will learn different ways to set up a Kubernetes cluster, including single-node as well as multi-node. We will also focus more on the Kubeadm way of running a multi-node cluster because the CKA certification expects you to troubleshoot Kubeadm cluster components and/or applications deployed on it.

Points to Remember

Kubernetes is a container orchestration software.

Kubernetes has a pluggable and extensible architecture.

Automated deployments, rollbacks, and autoscaling of apps are some of the key features of Kubernetes.

Multiple Choice Questions

Kubernetes project is maintained by

Google

Linux Foundation CNCF - Cloud Native Computing Foundation

Redhat

All of the above

What is CNI?

Custom Networking Interface

Container Network Interface

Client Network Interface

Cloud Network Interface

Which utility is used to interface with the Kubernetes cluster?

Docker CLI

Kubelet

Kubectl

Kube-proxy

Answers

b

b

c

Questions

Which is the smallest deployable unit of execution in Kubernetes?

Which controller creates a load balancer in the cloud infrastructure?

Keywords

Control Plane

API Controller

Scheduler

Cloud Controller Manager

Kube Controller Manager

ETCD Database

Kubelet

Kube-proxy

CNI

CRI

Further Reading

CRI (Container Runtime Interface): To know more about CRI, please browse this Kubernetes documentation page: (https://github.com/kubernetes/community/blob/master/contributors/devel/sig-node/container-runtime-interface.md)

CNI (Container Network Interface): To know more about CNI and various CNI plugins available, please browse this GitHub page: https://github.com/containernetworking/cni

CHAPTER 2

Installing Kubernetes

Introduction

In the previous chapter, we learned the benefits of Kubernetes. We also looked at the architectural components of Kubernetes and how they interact with each other to manage the cluster by providing high availability and scalability. Before we proceed and start experimenting with different examples, let us set up Kubernetes on your machine. There are many ways to set up the cluster, and you can choose the one that works best for you. However, it is recommended to try different options and observe how these tools set up Kubernetes.

In this book, all the examples are tried on the Kubernetes cluster installed on Ubuntu 22.04. In addition, any other tools, such as docker, minikube, microk8s, KIND, and kubeadm, are also installed on the same version of Ubuntu.

It is strongly suggested to use Ubuntu 22.04 as the base OS to run the experiments mentioned in this book. You may have MacOS or Windows, but please don’t worry because, by using Oracle VirtualBox and Vagrant, you can quickly get Ubuntu 22.04 up and running. In case you already have Ubuntu, you can still use the following approach (using Oracle VirtualBox and Vagrant) and carry out your Kubernetes learning in an isolated environment, separate from your main/host OS.

Structure

In this chapter, we will cover the following topics:

Basic Tools

Minikube

Microk8s

KIND - Kubernetes INside Docker

Kubeadm

Basic Tools

As mentioned earlier, using VirtualBox and Vagrant, you can set up a Kubernetes cluster using different methods and still keep your host OS isolated. You can either run multiple Kubernetes clusters isolated from each other using these basic tools or quickly clean up the installation in case of any system resource constraints.

Installing Oracle VirtualBox

Download the Oracle VirtualBox from https://www.virtualbox.org/wiki/Downloads and follow the instructions mentioned on the page for your OS.

Each command is a single line command and should be executed as such. If copying from the book and pasting the command, please ensure that there are no line breaks.

For an Ubuntu Linux machine, the steps are as follows:

wget https://download.virtualbox.org/virtualbox/7.0.6/

virtualbox-7.0_7.0.6-155176~Ubuntu~jammy_amd64.deb

sudo dpkg -i virtualbox-7.0_7.0.6-155176~Ubuntu~jammy_amd64.deb

sudo apt-get install -y make gcc perl

sudo apt-get install -y --fix-broken

sudo /sbin/vboxconfig

Installing Vagrant

You can download and install Vagrant using this link: https://developer.hashicorp.com/vagrant/downloads and follow the instructions mentioned on the page for your OS.

For an Ubuntu Linux machine, the steps are as follows:

wget -O- https://apt.releases.hashicorp.com/gpg | gpg --dearmor | sudo tee /usr/share/keyrings/hashicorp-archive-keyring.gpg

echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list

sudo apt update && sudo apt install vagrant -y

Note: In case you face any issues, either you can search for answers on Google or StackOverflow or raise an issue on this GitHub project: https://github.com/OrangeAVA/Ultimate-Certified-Kubernetes-Administrator-CKA-Certification-Guide/tree/main

Minikube

One of the easiest ways to set up a local Kubernetes cluster is by using Minikube. It works on Linux, MacOS, and Windows. The main highlights of Minikube are its ability to be deployed on VMs, containers, and even on bare-metal. When it comes to deploying on containers, it supports CRI-O, Docker, and containerd. Minikube also supports advanced features such as LoadBalancer, filesystem mounts, FeatureGates, and NetworkPolicy.

Since we have got Oracle VirtualBox and Vagrant setup done, we will launch Ubuntu 22.04 using Vagrant and then set up Minikube on it. Minikube supports containers, so we will use Docker to launch a single-node Kubernetes cluster.

Launching Ubuntu

No matter which OS you have, you can launch Ubuntu as mentioned earlier in this book. The following are the steps to launch it:

Create a folder named minikube in your home directory, enter it, and execute the following command:

rajesh@brainupgrade:~/minikube$ vagrant init bento/ubuntu-22.04 -m

This command will create the Vagrant file, named Vagrantfile, in the current folder. Modify this file and add the following content before the last line:

config.vm.provider "virtualbox" do |v|

v.memory=2048

v.cpus=2

end

Verify that the Vagrantfile content looks as follows:

Vagrant.configure("2") do |config|

config.vm.box = "bento/ubuntu-22.04"

config.vm.provider "virtualbox" do |v|

v.memory=2048

v.cpus=2

end

end

Now, you can run the following command to launch the Ubuntu OS:

rajesh@brainupgrade:~/minikube$ vagrant up

You can verify if the Ubuntu is in a running state using the following command:

rajesh@brainupgrade:~/minikube$ vagrant status

If you see an output similar to as shown in Figure 2.1, it means your Ubuntu OS is up and running:

Figure 2.1: Vagrant Minikube Ubuntu Setup

Installing Docker

Kubernetes supports many container runtimes that implement Container Runtime Interface (CRI). Docker is one of the prominent container runtimes with many useful tools, hence, it is preferred for our experiments.

Let’s follow the given steps:

Log in to the Ubuntu using the following command:

rajesh@brainupgrade:~/minikube$ vagrant ssh default

You will be greeted with a Welcome message.

Now, it is time to install Docker. Execute the following commands:

vagrant@vagrant:~$ sudo apt-get update && sudo apt-get install -y docker.io

vagrant@vagrant:~$ sudo usermod -aG docker $USER

vagrant@vagrant:~$ newgrp docker

You can verify if the installation is successful using the following command:

vagrant@vagrant:~$ docker ps

If you get no errors and single-line output, it means docker is working fine.

Installing Minikube

Now, download and install Minikube using the following commands:

vagrant@vagrant:~$ curl -LO  https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64

vagrant@vagrant:~$ sudo install minikube-linux-amd64 /usr/local/bin/minikube

Launching Kubernetes Cluster

To launch the cluster, docker can be used as the driver for Minikube. Refer to the following command:

vagrant@vagrant:~$ minikube start --driver=docker

Verify if the Minikube is in a running state using the following command:

vagrant@vagrant:~$ minikube status

Response to the preceding command would be similar to this output:

minikube

type: Control Plane

host: Running

kubelet: Running

apiserver: Running

kubeconfig: Configured

Testing the Minikube Kubernetes Cluster

To interact with the Kubernetes cluster, install the kubectl utility using the following commands:

vagrant@vagrant:~$ curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"

vagrant@vagrant:~$ sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

Let us find out the number of nodes using this kubectl utility:

vagrant@vagrant:~$ kubectl get nodes

You will get output similar to what is shown in Figure 2.2:

Figure 2.2: Minikube Cluster Nodes

We can now deploy a sample application, hello, as a test. The following commands and their outputs show how to deploy a sample app (kubectl run …) and also check if it is running (kubectl get …):

vagrant@vagrant:~$ kubectl run hello --image brainupgrade/hello

pod/hello created

vagrant@vagrant:~$ kubectl get pods

NAME    READY   STATUS    RESTARTS   AGE

hello   1/1     Running   0          12s

In the preceding snippet, the container image brainupgrade/hello (available at https://hub.docker.com/repository/docker/brainupgrade/hello/general) is used.

Cleaning Up Minikube Cluster

To destroy the Vagrant machine, run the following command:

rajesh@brainupgrade:~/minikube$ vagrant destroy

Microk8s

Microk8s is a lightweight, versatile, fully featured, conformant, secure, and multi-node Kubernetes system from Canonical company that powers Ubuntu OS.

To install microk8s, it is suggested to use a separate Vagrant machine and launch Ubuntu OS in a similar way to what is explained in the previous section (Minikube | Launching Ubuntu). Use a separate folder, say microk8s, to keep things clean and isolated. Here is a quick summary of the steps:

rajesh@brainupgrade:~/micr0k8s$ vagrant init bento/ubuntu-22.04 -m

Update Vagrantfile by specifying 2GB memory

rajesh@brainupgrade:~/micr0k8s$ vagrant up

Once Ubuntu OS is up and running, log in to the machine and install microk8s. Refer to the following four commands:

rajesh@brainupgrade:~/micr0k8s$ vagrant ssh default        ①

vagrant@vagrant:~$ sudo snap install microk8s --classic    ②

vagrant@vagrant:~$ sudo usermod -a -G microk8s vagrant     ③

vagrant@vagrant:~$ newgrp microk8s                         ④

Where:

① Command to log in to the Vagrant machine

② Command to install microk8s

③ Add the current user vagrant to the microk8s group

④ Reload the new microk8s group

Testing the Microk8s Kubernetes Cluster

Verify the cluster by first obtaining the nodes in the cluster using the following command:

vagrant@vagrant:~$ microk8s kubectl get nodes

The output of the preceding command would be similar to this:

NAME      STATUS   ROLES    AGE   VERSION

vagrant   Ready    <none>   20m   v1.26.3

If you set up the alias as shown here, otherwise, you need to prepend every kubectl command with microk8s:

vagrant@vagrant:~$ alias kubectl="microk8s kubectl"

Cleaning Up Microk8s Cluster

To destroy the Vagrant machine, run the following command:

rajesh@brainupgrade:~/microk8s$ vagrant destroy

KIND - Kubernetes INside Docker

The KIND tool was primarily designed to test Kubernetes itself as part of its automated testing process. It uses Docker containers as nodes and can be used for local development.

To install KIND, you can use a separate Vagrant machine and launch Ubuntu OS similar to what is explained in the earlier section (Minikube | Launching Ubuntu). Use a separate folder, say kind, to keep things clean and separate. Here is a quick summary of the steps:

rajesh@brainupgrade:~/kind$ vagrant init bento/ubuntu-22.04 -m

Update Vagrantfile by specifying 2GB memory

rajesh@brainupgrade:~/kind$ vagrant up

Once the Vagrant machine is up, log in using vagrant ssh default and install Docker. Refer to the following steps for reference:

rajesh@brainupgrade:~/kind$ vagrant ssh default

vagrant@vagrant:~$ sudo apt-get update && sudo apt-get install -y docker.io

vagrant@vagrant:~$ sudo usermod -aG docker $USER && newgrp docker

Now, install the KIND tool by executing the following commands:

vagrant@vagrant:~$ curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.17.0/kind-linux-amd64

vagrant@vagrant:~$ chmod +x ./kind

vagrant@vagrant:~$ sudo mv ./kind /usr/local/bin/kind

Using this KIND tool, you can launch the Kubernetes cluster as follows:

vagrant@vagrant:~$ kind create cluster --name kind

This command will create a single-node cluster immediately.

Testing the KIND Kubernetes Cluster

To interact with the Kubernetes cluster, install the kubectl utility using the following command:

vagrant@vagrant:~$ curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"

vagrant@vagrant:~$ sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

Let us find out the number of nodes using this kubectl utility:

vagrant@vagrant:~$ kubectl get nodes

You will get output similar to what is shown in Figure 2.3:

Figure 2.3: KIND Cluster Nodes

Multi-node KIND Cluster

You can use the config yaml file to create a multi-node cluster. Sample contents of the file are as shown here (save this to a file named cluster.yaml):

kind: Cluster

apiVersion: kind.x-k8s.io/v1alpha4

nodes:

- role: control-plane

- role: worker

- role: worker

Then, use the following command:

vagrant@vagrant:~$ kind create cluster --name mkind --config cluster.yaml

Where:

--name refers to the name of the cluster to be created.

You can verify the output by executing kubectl get nodes command, as follows:

Figure 2.4: KIND Multi-node Cluster

Cleaning Up KIND Cluster

To destroy the Vagrant machine, run the following command:

rajesh@brainupgrade:~/kind$ vagrant destroy

Kubeadm

Kubeadm is a tool designed to bootstrap Kubernetes clusters, and it is within the scope of the CKA certification. While you will not be expected to set up the cluster as part of the CKA exam, you will be expected to know how it works and troubleshoot any non-working Kubernetes parts and/or applications deployed on it.

There are several ways to install the cluster using kubeadm, such as:

Single-node cluster using a local machine.

Multi-node cluster using a group of local machines.

Multi-node cluster on a local machine using Vagrant and VirtualBox.

Since we have a versatile setup using Vagrant and VirtualBox, let us build a multi-node cluster on your host OS. To simplify the process, we have created two simple scripts:

Vagrantfile: Contains instructions on how to launch Vagrant Ubuntu machines.

common.sh: Contains config and tools setup instructions required on cluster nodes.

Launching Cluster Nodes

Using Vagrant, we can launch cluster nodes. Let us understand a few important statements in this file - Vagrantfile

NUM_WORKER_NODES=2  ①

IP_NW="10.0.0."     ②

IP_START=10

Vagrant.configure("2") do |config|

config.vm.provision "shell", env: {"IP_NW" => IP_NW, "IP_START" => IP_START}, inline: <<-SHELL

apt-get update -y

echo "$IP_NW$((IP_START)) master-node" >> /etc/hosts

echo "$IP_NW$((IP_START+1)) worker-node01" >> /etc/hosts

echo "$IP_NW$((IP_START+2)) worker-node02" >> /etc/hosts

SHELL

config.vm.box = "bento/ubuntu-22.04"     ③

config.vm.box_check_update = true

config.vm.define "master" do |master|

master.vm.hostname = "master-node"

master.vm.network "private_network", ip: IP_NW + "#{IP_START}"

master.vm.provider "virtualbox" do |vb|

vb.memory = 2048                     ④

vb.cpus = 2                          ⑤

end

master.vm.provision "shell", path: "common.sh"     ⑥

end

(1..NUM_WORKER_NODES).each do |i|

config.vm.define "node0#{i}" do |node|

node.vm.hostname = "worker-node0#{i}"

node.vm.network "private_network", ip: IP_NW + "#{IP_START + i}"

node.vm.provider "virtualbox" do |vb|

vb.memory = 2048

vb.cpus = 1

end

node.vm.provision "shell", path: "common.sh"        ⑥

end

end

end

Referring to the circled numbers in the preceding snippet, here are some notes about those instructions:

① Denotes the number of worker nodes. If your localhost machine has enough memory, say 16 GB RAM, then you can even have more than 2 worker nodes. However, if memory is limited, you can have one worker node as well. The preferred memory is 8GB of RAM or more to have a better functioning of a kubeadm-based Kubernetes cluster.

② Denotes that the IPs are assigned to the master as well as worker nodes, and the master IP starts with 10.0.0.10, followed by worker node IPs 10.0.0.11, 10.0.0.12, and so on.

③ Shows Ubuntu-22.04 as the base OS for this Kubernetes cluster.

④ Denotes the RAM to be allocated to the master node.

⑤ Refers to the CPU to be allocated to the master node.

⑥ Denotes that every node should have some basic OS configuration for kubeadm to function. The script, common.sh, contains these basic configurations besides having instructions to download and install required libraries such as kubeadm, kubelet, kubectl, containerd - container runtime.

You can create a separate folder, say kubeadm, and copy the contents of Vagrantfile and common.sh files. These files are kept here https://github.com/OrangeAVA/Ultimate-Certified-Kubernetes-Administrator-CKA-Certification-Guide/tree/main/chapter_02

To launch cluster nodes (master as well as worker nodes), execute the following Vagrant command:

rajesh@brainupgrade:~/kubeadm$ vagrant up

You can verify if all the cluster nodes are up and running by using the following command, as shown in Figure 2.5:

Figure 2.5: Kubeadm Machines

Initializing the Master/Control Plane Node

To launch the Kubernetes control plane, first log in to the master node by using the following command:

rajesh@brainupgrade:~/kubeadm$ vagrant ssh master

Then, initialize the Kubernetes using the kubeadm init command by specifying the master node IP as apiserver and pod IP allocation range:

vagrant@master-node:~$ sudo kubeadm init --apiserver-advertise-address 10.0.0.10 --pod-network-cidr 192.168.0.1/16

This step may take a while because it downloads Kubernetes control plane component images and then launches them. Once this command is executed successfully, it prints out two important notes:

Instructions on how to use the cluster, which you should execute immediately (as follows):

mkdir -p $HOME/.kube

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

sudo chown $(id -u):$(id -g) $HOME/.kube/config

Command to be used from worker nodes to join them to the cluster. You will see the output similar to the following, but the token and hash would be different in your case:

kubeadm join 10.0.0.10:6443 --token bfygwu.vgzpweo3bc17aapo \ --discovery-token-ca-cert-hash

sha256:572b5df7183b901a1fa90e5f516574f03d5bd6a5eeb5e1c f77997361986b2c8a

You can copy the kubeadm join command from your console and execute it on the other two worker nodes. To log in to the worker node 1, use the following command:

rajesh@brainupgrade:~/kubeadm$ vagrant ssh node01

Then, add sudo before executing the kubeadm join command as follows:

vagrant@worker-node01:~$ sudo kubeadm join 10.0.0.10:6443 --token bfygwu.vgzpweo3bc17aapo \

--discovery-token-ca-cert-hash sha256:572b5df7183b901a1fa90e5f516574f03d5bd6a5eeb5e1cf77997361986b2c8a

Once the preceding command is successfully executed, you will see an output similar to this:

This node has joined the cluster:

* Certificate signing request was sent to apiserver and a response was received.

* The Kubelet was informed of the new secure connection details.

Follow the same procedure to join worker-node02 to the cluster. When done, issue the following command on the master node to verify that all nodes are part of the cluster and are in a running state:

Figure 2.6: Kubeadm Cluster Nodes

As you can see, all the worker nodes have joined the cluster and every node is in a running state. You can also see the control plane components and where those are deployed. Refer to the following command and its output, as shown in Figure 2.7:

Figure 2.7: Control Plane Components

Control plane components are deployed on the master-node and grouped under the kube-system namespace (denoted by -n kube-system in the command as a filter). In Kubernetes, the namespace is used to logically group various Kubernetes resources and thus isolate them from the rest. There could be other components deployed and grouped under the kube-system namespace. Hence, to filter out only the control plane components, the -l tier=control-plane filter is used, meaning components labeled as tier=control-plane. To print IP and node columns, a wide output option (denoted by -owide) is used. In the next chapter, we will look at such kubectl command options in detail.

It is important to note that kubeadm uses the /etc/kubernetes directory to keep config files, static pod manifests, certificates, and key files.

Note: Kubelet launches PODs from definitions found in the manifests folder and/or received from the API-server itself.

Kubelet-related files are kept under /var/lib/kubelet folder. This folder contains the kubelet config, runtime environment flags, certificate, and key files. Kubelet uses the static pod manifest path mentioned in its config file, which is kept at /var/lib/kubelet/config.yaml, to launch control plane components.

Refer to the following content of the manifests folder:

vagrant@master-node:/etc/kubernetes/manifests$ ls

etcd.yaml  kube-apiserver.yaml  kube-controller-manager.yaml  kube-scheduler.yaml

If you inspect these files, you will notice that all control plane components are defined as Kubernetes PODs themselves in YAML format, and kubelet uses these definitions to launch the Kubernetes control plane.

Tip: If the cluster is not working, then you are likely to spot the problem in these manifest files or the kubelet config itself (an important point from the certification exam perspective).

Testing the Kubeadm Kubernetes Cluster

To interact with the Kubernetes cluster, log in to the control plane using the following command:

rajesh@brainupgrade:~/kubeadm$ vagrant ssh master

Deploy the hello app using the following command:

vagrant@master-node:~$ kubectl run hello --image brainupgrade/hello

You will get output similar to the following:

vagrant@master-node:~$ kubectl get pods

NAME    READY   STATUS    RESTARTS   AGE

hello   1/1     Running   0          63s

Cleaning Up the Kubeadm Cluster

To destroy the Vagrant machines, run the following command:

rajesh@brainupgrade:~/kubeadm$ vagrant destroy

Conclusion

In this chapter, we explored four ways to install the Kubernetes cluster. To keep things clean and isolated from your host OS, we used Vagrant and VirtualBox tools. You can set up the cluster the way you find suitable for your condition; however, from the certification exam perspective, you are expected to be well versed with the kubeadm way of setting up the cluster and troubleshooting it. Troubleshooting becomes a key part of the exam.

In the next chapter, we will look at different workload types and use cases associated with them.

Points to Remember

Kubeadm provides easy methods to initialize the control plane and join other nodes to the cluster.

KIND uses Docker containers as Kubernetes nodes in the cluster.

Microk8s is the lightweight, fast, conformant, and versatile utility to launch the Kubernetes cluster.

Minikube can be used with Docker or VM to launch the Kubernetes cluster.

Multiple Choice Questions

API-server certificate and key are kept in which folder?

/var/lib/kubernetes

/etc/kubernetes/api-server

/etc/kubernetes/pki

None of the above

API-server connects to ETCD running at which location?

https://127.0.0.1:2379

https://127.0.0.1:2380

https://10.0.0.10:443

https://10.0.0.10:6443

API-server runs on which port?

6443

443

8080

None of the above

Which data directory is used by ETCD?

/var/lib/etcd

/var/data

/etc/kubernetes/etcd/data

None of the above

ETCD server uses its certificate and key files located in which folder?

/var/lib/etcd/pki

/var/lib/etcd

/etc/kubernetes/etcd

/etc/kubernetes/pki/etcd

Which cluster CIDR is used?

192.168.0.1/16

192.168.0.1/8

192.168.0.1/12

10.96.0.0/12

Which range is used for service cluster IPs?

10.96.0.0/16

10.96.0.0/12

10.96.0.0/8

None of the above

At which location Scheduler config file is kept?

/etc/kubernetes

/var/lib/scheduler

/var/lib/kubelet

/etc/kubernetes/scheduler

Answers

c

a

a

a

d

a

b

a

Questions

Does kubelet launch control plane components?

Where does kubeadm keep static pod definition files?

Keywords

Minikube

Microk8s

KIND

Kubeadm

Static Pod

CHAPTER 3

Workload Objects – Pod, Deploy, StatefulSet

Introduction

In the previous chapter, we learned multiple different ways to set up a Kubernetes cluster. In this chapter, we will understand how to deploy applications in the Kubernetes cluster using the workload objects, namely Pod, Deploy (or Deployment), and StatefulSet. The majority of the applications in the Kubernetes cluster are deployed using these workload objects.

This chapter also focuses on the containerProbes, which are important to improve overall application responsiveness and thus availability.

Examples referred to in this chapter are run in the cluster that is set up using KIND.

Structure

In this chapter, we will cover the following topics:

Pod

Deployment

StatefulSet

Pod

A Pod is a group of one or more containers with a shared network and storage. It also contains specifications for how to run the containers.

Pods are the smallest executable units in Kubernetes. All contents of a Pod are always co-located. Since the network is shared by all the containers within a Pod, it is a kind of ‘localhost’ to the container processes.

A Pod can also contain init containers that run during Pod startup and before application/app containers. Init containers can contain setup scripts or utilities meant to initialize things required for app containers. For debugging purposes, one can inject ephemeral containers (temporary containers) in an existing Pod.

Running Pods

The quickest way to run a Pod is by using an imperative command:

kubectl run hello --image docker.io/brainupgrade/hello:1.0

In this command, hello is the name of the Pod to be run, and the --image option is used to specify the container image to be used, that is, docker.io/brainupgrade/hello:1.0

Where:

docker.io is the image repository

brainupgrade is the author id/account name on docker.io

hello is the name of the image

1.0 is the tag or version of the image

To verify if the Pod is running successfully, the following command can be used:

kubectl get pods

The following command and output are given for reference.

Figure 3.1: List of Pods - Hello

Another way to run a Pod is using a YAML file. Refer to the following example:

Figure 3.2: Example - Hello Pod Specification

The preceding example follows the YAML (YAML Ain’t Markup Language) format and contains minimal specifications required to run the Pod. In line number 6, you can see the containers section under which you can specify one or more containers. To run the Pod specified in the YAML, as shown in Figure 3.2, execute the following command:

kubectl apply -f https://raw.githubusercontent.com/brainupgrade-in/book-kubernetesadmin/main/chapter_03/pod-hello.yaml

You can launch Pods using any of the images available in the image repositories, public or private, and if private, then you can use registry credentials, which will be explained later.

Let us launch one more Pod using the image docker.io/brainupgrade/tshoot. This image contains a few troubleshooting utilities, such as curl, bash, and so on, and is thus useful during problem investigation. Use the following command to launch this troubleshooting Pod:

kubectl run tshoot --image docker.io/brainupgrade/tshoot

You can verify if the Pod is running (kubectl get pods), and if yes, use the following command to enter inside the container of tshoot Pod:

kubectl exec -it tshoot -- bash

Refer to Figure 3.3 for reference. Type exit and enter to come out of the container:

Figure 3.3: Pod - tshoot

Restart Policy

By default, all containers will always be restarted, regardless of whether the containers exit successfully or not. To change this default behavior, one can set the Pod restartPolicy field with these possible values: OnFailure or Never; otherwise, it will be set as Always.

This applies to all the containers in the Pod. Kubelet attempts to restart containers with an exponential back-off delay (10s, 20s, 40s…) capped at five minutes and on the same node.

To know the restart policy setting of the Pod containers, use the following command:

kubectl get pods hello -o jsonpath=’{@.spec.restartPolicy}{“\n”}’

The result of the kubectl command can be filtered, and in the preceding scenario, JSON path is used to filter the restartPolicy field value.

Pod Lifecycle

A high-level summary of where a Pod is in its lifecycle is denoted by the Pod phase. Here are the possible values for the phase:

Pending

This means one or more containers are yet to be set up and made ready to run. This also includes time when Pod is waiting to be scheduled besides time spent downloading container images.

Running

The Pod is bound to the node and all the containers are created. At least one container is started or is being started.

Succeeded

All containers in the Pod have exited or terminated successfully.

Failed

At least one container terminated in failure, while the rest have terminated.

Unknown

The state of the Pod could not be obtained because of network issues.

Pod specification includes a status field, which contains a phase field. It also contains the statuses of all the containers.

Container Probes

A probe is a diagnostic performed by Kubelet on a container periodically. The aim of conducting a probe is to find out if the application inside the container is started; the container is running, and ready to respond.

startupProbe

This indicates whether the application within the container is started or not. Until this probe succeeds, all other probes if defined are disabled. If this probe fails, the Kubelet kills the container and makes a container restart attempt based on the restart policy.

This probe is generally used when the application requires some time to initialize, such as by loading large datasets, and so on. Refer to Figure 3.4 for reference:

Figure 3.4: Probe - startup

In the preceding example, a few options are configured under startupProbe. The probe mechanism used in this case is httpGet. Other mechanisms include exec, wherein Kubelet executes a command inside the container, tcpSocket - wherein Kubelet makes a network call to check if the mentioned socket is open, and gRPC - wherein Kubelet performs a remote procedure call using gRPC (https://grpc.io/).

Apart from the probe mechanism, one can define thresholds to determine the success and failure of the probe attempt and the period between consecutive probe attempts.

readinessProbe

This probe indicates if the container is ready to respond to requests. Failure of this probe results in disabling of routing of any more requests to this container till this probe succeeds.

This probe is generally used when the application is temporarily busy responding to current requests and thus unable to serve any more new requests, for example, loading large data, processing heavy images, and so on. Refer to Figure 3.5 as a reference example:

Figure 3.5: Probe - readiness

livenessProbe

This probe indicates if the container is running. If this probe fails, the Kubelet kills the container and restarts it, subject to the restart policy. This probe intends to discover applications that are in a deadlock or hung state.

The probes, readiness, and liveness, run in parallel, hence, please ensure to have a longer probe interval for liveness than readiness to have the desired effect.

Figure 3.6: Probe - liveness

Deployment/Deploy

Pods deployed using only a Pod spec do not get automatic healing (also known as self-healing) behavior when terminated by the Kubelet due to node capacity constraint or if the node on which it is running goes down. Additionally, Pods cannot be scaled when created using a Pod template. To get self-healing and replication functionality (scale in and scale out), Pods are deployed using the Deployment (also known as Deploy) template.

Deployment controllers manage the resources created using the Deployment template and create one ReplicaSet controller for each version of the deployment resource created. Refer to the following example of deployment:

Figure 3.7: Deploy - app (line numbers enabled)

Figure 3.7 shows a Deployment named app having Pod specification embedded under the template section (line #12). The number of Pod replicas is specified using the field replicas (line #8).

ReplicaSet controller associated with this Deployment resource manages replicas for the Pods having label(s) app: app (key: value style) defined under the selector section (line #11 in this case). So, if Pod labels (line #15) don’t match the ones defined under the selector section, then the ReplicaSet controller won’t manage the replication for the Pod defined in this template.

Let us run the example mentioned earlier using the imperative command as follows:

kubectl create deploy app --image brainupgrade/hello:1.0

When executed, Kubernetes creates deployment, ReplicaSet, and Pod resources and labels them with the key as app and value as app, that is, the name of the deployment resource. To list all the resources created by the preceding command, refer to Figure 3.8:

Figure 3.8: App Resources

As you can see in Figure 3.8, the deployment controller created a deployment resource, one replicaSet, and this replicaSet created one Pod replica. The name of the Pod has replicaSet name as its prefix.

Scalability

By default, one Pod replica is created by the associated replicaSet. To cater to a higher load on the application, more replicas can be created. One can use the manual command to scale up application replicas, as follows:

kubectl scale --replicas 3 deploy app

Refer to Figure 3.9 for reference:

Figure 3.9: Scaling the App

As you can see in Figure 3.9, the ReplicaSet created 3 Pod replicas. You can get more information about this deployment using the kubectl describe command. Refer to the following reference:

Figure 3.10: Describe Deploy

Rollout of a New Change

Using Deployment, a new version of the application can be easily rolled out. If labels, container images, or their resources are updated, then a rollout is triggered. Using an imperative command, as shown in Figure 3.11, a new rollout is triggered.

Figure 3.11: Rollout of New Change

As part of the new change, version 2.0 of the hello container image is set. Thus, a new replicaSet (app-6756666d57) is created, which manages the replicas for version 2.0 of the deployment. This new replicaSet (app-6756666d57) created 3 replicas as expected, and the old replicaSet (app-6f98595b6) removed all the replicas associated with the old version of the app.

Rollback

In case the new change fails to load the app and doesn’t bring the desired effect for whatever reasons, you may want to roll back the change. Now, imagine if you have a large number of replicas that are spread across multiple nodes, doing manual rollback will not only be cumbersome but can also cause greater damage. Kubernetes provides automated rollback as well, and using the following command, rollback can be done:

kubectl rollout undo deploy app

To verify if rollback is successfully done, you can use the following command:

Figure 3.12: Rollback the Change

The output will be similar to the one shown in Figure 3.12. Deployment refers to version 1.0 of the app, and the replicaSet associated with version 1.0 created 3 replicas. The replicaSet associated with version 2.0 removed all the replicas.

If there are multiple revisions available, then to roll back to any particular revision, say 1 in this case, the following command can be used:

kubectl rollout undo deploy app --to-revision 1

If the number of replicas to be rolled out is large in numbers and you want to pause the deployment to correct any deployment issues, you can execute the pause command as follows:

kubectl rollout pause deploy app

And once all things are OK, you can resume it by using the following command:

kubectl rollout resume deploy app

Deployment controller creates or destroys replicas in parallel during scale-up or scale-down operations. It is also stateless because it does not maintain any state for the replicas, assuming they are transient.

StatefulSet

Applications that require stable persistent storage, like clustered databases, are best suited to use StatefulSet. StatefulSet guarantees that the same storage is attached to the replica even if it is re-launched or destroyed and created again. It also provides ordered scaling up and down and ordered automated rollout. Each Pod replica is numbered and unique, thus not interchangeable because each one has its persistence storage associated with it and a unique network identifier.

Limitations

The following things should be kept in mind while using StatefulSet:

The storage required for a given Pod must be manually or dynamically provisioned.

Storage/volume associated with StatefulSet may not be deleted when StatefulSet is deleted or even scaled down.

To get network identity for the Pods, you need to create a Headless Service. In the next chapter, we will look at how to create such a service.

Running StatefulSet - Postgre Database

Let us use StatefulSet to launch the Postgre database. Refer to the YAML snippet in Figure 3.13.

Figure 3.13: STS - Postgre Example

As you can see in the preceding example, volume information is provided through the element volumeClaimTemplates. Full YAML code that includes Service definition can be found at https://raw.githubusercontent.com/brainupgrade-in/book-kubernetesadmin/main/chapter_03/sts-postgre.yaml. To run the example, you can use the following command:

kubectl apply -f https://raw.githubusercontent.com/brainupgrade-in/book-kubernetesadmin/main/chapter_03/sts-postgre.yaml

Kubernetes will immediately create the StatefulSet (sts is the short form) resource and shortly aftre, the Pod for this StatefulSet. To scale up the replicas, run the following command:

kubectl scale --replicas 3 sts postgre

To observe the orderly Pod creation, you can use another terminal and issue the following command with the watch option denoted by -w at the end:

kubectl get pods -l app=postgre -w

Refer to Figure 3.14 for reference:

Figure 3.14: Postgre Resources - Terminal View

Here, persistent volumes are automatically created based on the volumeClaimTemplates section defined in the StatefulSet YAML template. In the chapter related to Volume, we will look at how to set up an auto-volume provisioner.

To list the volumes associated with the Pod replicas, you can run the following command:

kubectl get pv (pv stands for persistentvolume)

And the output will be similar to the one shown in Figure 3.15:

Figure 3.15: STS Postgre Volumes

Conclusion

In this chapter, we learned about the Pod and its lifecycle. We also explored different container probes to improve the overall availability of the application. To reiterate, the startupProbe is executed at the beginning when the container is created, and until it succeeds, the other two probes - readinessProbe and livenessProbe - will not be attempted. Readiness and liveness probes are attempted in parallel; however, the container will be restarted if the livenessProbe fails. However, if the readinessProbe fails, Kubernetes will not route requests to this Pod.

In the enterprise contexts, stateless applications are often deployed using Deployment and StatefulSet for stateful applications. In the case of Deployment, scale up and down happens in parallel, while for StatefulSet, Kubernetes guarantees orderly scale up and down besides unique persistence storage to each replica, which is not interchangeable as opposed to Deployment replica.

In the next chapter, we will look at how to expose the apps within the cluster and outside the cluster, including exposing them over the Internet.

Points to Remember

Pods are often deployed using higher-level specifications, like Deployment, StatefulSet, and so on, to achieve scalability and auto-healing.

Container probes are important to improve overall application responsiveness.

Deployment is used for stateless applications and scaling of replicas happens in parallel.

When apps need stable persistent storage, StatefulSet can be used.

Multiple Choice Questions

Which one of the following is not the lifecycle phase of a Pod?

Running

Successful

Pending

Unknown

To notify when the application is busy and can’t process new requests, which container probe should be used?

livenessProbe

readinessProbe

startupProbe

None of the above

Which container probe should be used to detect application deadlocks?

startupProbe

readinessProbe

livenessProbe

None of the above

For legacy applications that take a good amount of time to boot up, which probe is more appropriate?

livenessProbe

startupProbe

readinessProbe

None of the above

What is the default restartPolicy configured for Pod?

Always

Never

OnFailure

None

Which one of the following is not a feature of Deployment?

Fast rollout because scaling happens in parallel

Pod replicas are interchangeable

Stable network identifier

One replicaSet for every unique version of Deployment

Which one of the following is not a feature of StatefulSet?

Stable network identifier

Unique persistent storage

One replicaSet for every unique version of Deployment

Non-interchangeable Pod replicas

When the StatefulSet is scaled down, corresponding persistent volumes of those replicas are also removed.

True

False

Answers

b

b

c

b

a

c

c

b

Questions

How to roll back a Deployment to a particular revision number?

When rolling out a new change using StatefulSet, which replica will be updated first?

Keywords

Pod

Deployment

StatefulSet

Container Probes: startupProbe, readinessProbe, livenessProbe

Automated rolling updates

CHAPTER 4

Service and Ingress - Exposing Apps Outside the Cluster

Introduction

In the previous chapter, we learned how to deploy an application using workload objects like Pod, Deploy, and StatefulSet.

In this chapter, we will learn how to expose the deployed applications both within the cluster and outside the cluster. We will also see how to use an Ingress Controller to publish applications to the internet.

Examples referred to in this chapter and thereafter are run in the cluster that is set up using KIND, unless specified otherwise.

Structure

In this chapter, we will cover the following topics:

Service

Ingress

Service

Pods are ephemeral and may be terminated and launched on some other node. The IP allocated to a Pod is dynamic; hence, accessing the application by referring to the Pod IP is not desirable. Additionally, an application may have multiple Pod replicas running to cater high load on the application, and using their IPs to communicate is a brittle approach that is sure to fail.

Kubernetes Service provides a network URL, a DNS name, to a set of Pod replicas, thus decoupling the application from its callers. It also provides load balancing if multiple replicas of the application are running.

Creating a Service

Following is the YAML snippet for the request-logger application. This application logs the request details of every caller and returns all the calls logged so far as a response:

Figure 4.1: Logger App

To run this app, execute the following command:

kubectl apply -f https://raw.githubusercontent.com/brainupgrade-in/book-kubernetesadmin/main/chapter_04/logger-deploy.yaml

Kubernetes immediately creates a deployment resource named logger and also spawns one Pod replica as specified in the YAML template. Now in order to expose this application as a network application having its own DNS name, you can create a service.

You can use the imperative command as follows to expose this app as a Service:

kubectl expose deploy logger --port 80 --target-port 8080 --name logger

Where:

--port 80 is the service port on which caller requests are received.

--target-port 8080 is the container port on the target pod replicas also known as endpoints.

--name logger is the name of the service.

A YAML template can also be used to create this service. Refer to the following YAML template for reference:

Figure 4.2: Logger Service

If the Service is created using an imperative command, then Kubernetes automatically sets the selector labels matching with Pod labels as defined in the Deployment specification. The Selector having matching labels as that of Pod labels is the key connection between the Service and its Endpoints (Pod replicas).

To verify if the app can be accessed using Cluster DNS name, we can perform a quick test using a publicly available docker image (brainupgrade/tshoot) that is useful in doing quick troubleshooting.

Launch the troubleshooting Pod using the following command:

kubectl run tshoot --image brainupgrade/tshoot

Once the Pod is running, enter inside this Pod’s container using the following command, and you will be on the bash command prompt:

kubectl exec -it tshoot -- bash

This tshoot Pod has many debugging tools, including curl. Using curl, you can check if the app responds to the web request on service port 80.

curl logger:80

You can also use the full DNS URL by adding a namespace and other info as follows:

curl logger.default.svc.cluster.local:80

You will see an output similar to that shown in Figure 4.3:

Figure 4.3: Verify App Using Service URL

Kubernetes logger service receives the request on port 80 and redirects it to port 8080 on the target endpoints where the application is running on port 8080. The application responds with JSON output and the same can be viewed on the terminal.

The service created in the preceding example is visible within the cluster only, and it is the default service type. Following is the list of types of Services supported by Kubernetes:

ClusterIP

NodePort

LoadBalancer

ExternalName

ClusterIP

If the Service type is not defined at the time of Service creation, the default Service type is created. The scope of this Service is limited to visibility within the cluster. Kubernetes assigns an IP to each Service, which is accessible only inside the cluster.

To facilitate communication between the frontend app and the backend app, this type of Service is typically used. The Frontend app does not need to know which replica is currently servicing the backend since the backend is abstracted using the Service URL. The range of service IPs that the Kubernetes cluster can use is specified at the time of cluster creation using the option service-cluster-ip-range.

In case you don’t need a load balancing feature, then you can set .spec.clusterIP to None and Kubernetes will not assign any ClusterIP for your Service. This type of Service is also known as Headless Service, and such a Service is useful for applications like Cassandra clustered databases.

To know the ClusterIP assigned to the logger Service, you can execute the following command:

kubectl get svc logger

NodePort

If the application is required to be accessible outside the cluster but within the network same as that of the cluster, NodePort service can be used. Kubernetes control plane allocates a port from a range specified by the option --service-node-port-range at the time of cluster creation. The default range is between 30000 and 32767. Every node in the cluster listens on that assigned port and forwards the traffic to one of the ready Service endpoints. Kubernetes also creates a ClusterIP and port for every NodePort service.

Running a NodePort Service

First, let us deploy the hello application and then create a NodePort service named hello using the following commands:

kubectl create deploy hello --image brainupgrade/hello:1.0

kubectl create svc nodeport hello --tcp=80:8080

In the create svc command, --tcp flag contains the service port (80) and the target endpoint port (8080).

You can find out the NodePort assigned for this service using the following command:

kubectl get svc -l app=hello

Output will be similar as follows:

NAME    TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE

hello   NodePort   10.96.65.121   <none>        80:32315/TCP   57m

As you can see from the output, the NodePort assigned to this service is 32315, and the cluster port is 80, which is accessible within the cluster at IP 10.96.65.121.

Before we perform a few more checks, let us find out the IPs of the cluster nodes using the following command. Flag -owide is added to get wide output, that is, additional info using more columns.

kubectl get nodes -owide

Figure 4.4: List of Nodes with IP

Now access the service using node port and cluster nodes. You will see a response similar to the one shown in Figure 4.5:

Figure 4.5: Service Access Using NodePort

As you can see in the preceding figure, no matter which node receives the request, the request is redirected to the available endpoint by the kube-proxy running on that node.

Clean Up

To remove the service created above, run the following command:

kubectl delete svc hello

LoadBalancer

Applications can be exposed to the internet or a wide area network using the Service type LoadBalancer. Kubernetes creates the configuration similarly to how it does for NodePort service, and the cloud-controller-manager asynchronously creates a cloud provider-specific load balancer and publishes it under the .status.loadBalancer field of the Service object.

Running a LoadBalancer Service

To make the LoadBalancer feature work for the KIND-based cluster, we need to install a third-party controller and run the following two commands:

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.7/config/manifests/metallb-native.yaml

kubectl apply -f https://raw.githubusercontent.com/brainupgrade-in/book-kubernetesadmin/main/chapter_04/logger-deploy.yaml

Now, you can launch the LoadBalancer service using the following command:

kubectl create svc loadbalancer hello --tcp=80:8080

Once the Service is created, you can obtain the LoadBalancer IP/URL using the following command. Under the EXTERNAL-IP column, LoadBalancer IP is printed:

kubectl get svc hello -owide

To test the application using this LoadBalancer IP, run the following command:

curl 172.18.255.200:80

Refer to Figure 4.6 for your reference. If the LoadBalancer IP (under the output column EXTERNAL-IP) is different in your case, modify the preceding command appropriately.

Figure 4.6: LoadBalancer Service - hello

In the case of a Cloud-based Kubernetes cluster, traffic from the cloud load balancer is directed at the backend Pods, and the cloud provider decides how the load is balanced.

ExternalName

This type of Service is typically used when applications deployed in the Kubernetes cluster want to access the application currently residing outside the cluster and may be migrated in the future. Refer to the following YAML snippet for example:

Figure 4.7: ExternalName Service

In the preceding example, applications within the cluster can access the database using this Service name, and the cluster DNS Service returns a CNAME record with the value app-db.global-bank.com. In the future, if app-db is migrated into the Kubernetes cluster, then this Service type needs to be changed, and appropriate selectors need to be added without impacting the caller.

Note: The field externalName accepts an IPv4 address string as a DNS name; however, external names that resemble IPv4 addresses are not resolved by DNS servers.

Tip: Do not use IPv4 as an external name, instead set the CNAME value.

Ingress

Ingress routes HTTP and HTTPS requests from outside the cluster to the services running inside the cluster. To achieve this, an Ingress Controller is used in conjunction with routing rules embedded in Ingress, an API object.

When a Load Balancer service is used, Kubernetes creates a cloud provider-specific load balancer (LB). While load balancers are highly performant, they come at a cost. Therefore, creating one load balancer per app service is not a good economic choice. One Load Balancer per cluster serves the purpose in most cases, and using an Ingress controller to route requests for many domains served by the apps deployed in the Kubernetes cluster is the recommended approach.

Installing Ingress Controller

There are many Ingress Controllers available. Some of them are listed at (https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/). You can run the following command to install the Ingress Controller for your KIND cluster:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/baremetal/deploy.yaml

Ingress controller, when installed in a cloud-based Kubernetes cluster, also creates a cloud load balancer specific to the cloud service provider. Cloud load balancer runs outside the Kubernetes cluster and routes the requests to the Ingress controller, which runs inside the cluster. It uses the node ports allocated to the Ingress Controller Service on the cluster nodes. Few advanced Ingress controllers support requests other than HTTP and HTTPS. For example, Kong Ingress Controller (docs.konghq.com/kubernetes-ingress-controller/latest/) can be used to route TCP/UDP socket requests too.

Running the Hello App

Run the hello app if it’s not already running using the following command:

kubectl create deploy hello --image brainupgrade/hello:1.0

If the Service for this app does not exist, then you can create it using the following command:

kubectl create svc clusterip hello --tcp=80:8080

Once the hello app is running, including its Service, we can publish it to the internet by creating an Ingress resource/object. An Ingress holds the mapping rules of Host URLs, paths, and corresponding Services. Refer to the following YAML code:

Figure 4.8: Ingress Rules - path mappings

As you can see in Figure 4.8, the hello service is mapped to the domain hello.brainupgrade.in, and based on this Ingress resource, the Nginx Ingress controller routes the external requests coming for this domain to one of the available Service endpoints.

Using the following command, you can create the preceding Ingress resource:

kubectl apply -f https://raw.githubusercontent.com/brainupgrade-in/book-kubernetesadmin/main/chapter_04/ingress-hello.yaml

Once this Ingress resource is created, the Ingress controller associates the load balancer URL provided by the cloud service provider, or else to one of the cluster node addresses. Refer to Figure 4.9 for reference:

Figure 4.9: Ingress - hello

Setting Up Load Balancer - Nginx

Most of the cloud load balancers are based on the Nginx proxy feature and fine-tuned for high performance. Even though ours is a local Kubernetes cluster, we can still use the Nginx server as load balancer software to route outside cluster requests coming for this domain to the Ingress controller running inside the Kubernetes cluster.

Using the following commands, we can quickly set up the Nginx on the Vagrant machine:

sudo apt install nginx -y

Once nginx is installed, then configure the routing information. Refer to Figure 4.10 for reference:

Figure 4.10: Nginx Load Balancer proxy config

You can add the preceding config using the following command on the vagrant machine:

sudo wget https://raw.githubusercontent.com/brainupgrade-in/book-kubernetesadmin/main/chapter_04/kubernetes.conf -O /etc/nginx/sites-available/kubernetes.conf

Edit this conf file using sudo vi /etc/nginx/sites-available/kubernetes.conf and fill up the ingress address and nodeport information on line #2. You can get these details using the following commands, as shown in Figure 4.11:

Figure 4.11: Ingress Address and Ingress Nginx Controller NodePort

This is how line #2 may appear in the conf file. Make appropriate modifications at your side.

server       172.18.0.2:31343;

Once the conf is updated, enable the conf and restart the Nginx server using the following commands:

sudo ln -s /etc/nginx/sites-available/kubernetes.conf /etc/nginx/sites-enabled/

sudo systemctl restart nginx

Now a minor DNS tweak on the local vagrant machine is needed so that requests initiated from this vagrant machine to the hello app URL can be handled by the Nginx Load Balancer. Just add the app URL in the /etc/hosts file. Refer to the following code for reference:

sudo vi /etc/hosts

And append the app URL at the end of first-line as follows:

127.0.0.1 localhost hello.brainupgrade.in

For cloud-based clusters, you need to update DNS records by adding CNAME as the domain name and its value as the load balancer address.

Testing the App URL

On the vagrant machine, run the curl command by giving the app URL as an input, and you can expect a response from the available app replica. Refer to Figure 4.2 for reference:

Figure 4.12: Testing App Using its URL

Interaction Summary - Ingress

To make things clear on the overall interaction among various components, you can refer to Figure 4.13:

Figure 4.13: Ingress Routing

Conclusion

In this chapter, we learned how to expose the app within the cluster using the ClusterIP Service, thus decoupling it from its callers. We also explored different types of Service, such as NodePort, to make the app accessible from outside the cluster but from the same network in which the cluster resides. For external requests, especially from the Internet, the LoadBalancer Service can be used. Apps that are yet to be migrated to Kubernetes can be referred using the ExternalName Service, so that clients don’t need to be changed after their migration.

We also learned a better way to publish apps to the internet and route the requests using the Ingress controller, instead of using a LoadBalancer for each app service.

In the next chapter, we will look at the scaling of stateless apps in detail. We will also explore ways to set the scaling thresholds to automatically scale up as well as down the apps.

Points to Remember

ClusterIP is the default Service type created by Kubernetes.

NodePort Service creation also includes the creation of ClusterIP Service.

HeadlessService does not provide a load-balancing feature; however, it provides an option to use a custom discovery protocol.

The Ingress controller component makes use of Ingress rules defined using Ingress objects to route requests received for domains served by the apps.

Multiple Choice Questions

Which one of the following is not a Service type?

ClusterIP

NodePort

Headless

ExternalName

LoadBalancer

When the NodePort service type is used, Kubernetes creates a node port on which node(s)?

Control plane node

Worker nodes

First worker node only

All the nodes

Which Service type should be used to connect to the app which is yet to be migrated to the Kubernetes cluster?

NodePort

Headless

ClusterIP

ExternalName

Which Service URL can be used to access a Service named hello in the default namespace? Select the best answer.

hello

hello.default.svc

hello.default.svc.cluster.local

All of the above

To make sure connections from a particular client are passed on to the same Pod, which Service attribute should be set?

.spec.stickiness: ClientIP

.spec.sessionAffinity: ClientIP

.spec.internalTrafficPolicy: None

.spec.sessionAffinityConfig.clientIP: true

Which one of the following statements is not true?

Ingress does not support specifying TLS for app host URLs

Multiple hosts can be specified as part of the Ingress resource

Ingress controller drops TLS at the Ingress point

Ingress can just contain the defaultBackend spec without any rules

Ingress controller can support routing HTTP traffic to multiple host names at the same IP address.

True

False

For a single host, multiple Services can be configured specifying one Service per path as part of the Ingress definition.

True

False

Answers

c

d

d

d

b

a

a

a

Questions

Does ExternalName service use selectors?

Can the Service have multiple ports?

Keywords

ClusterIP

NodePort

LoadBalancer

Ingress Controller

Ingress

Headless Service

ExternalName Service

CHAPTER 5

Deploy and Scale - Stateless Apps

Introduction

In the previous chapter, we learned how to expose applications both within and outside the cluster so that callers of the applications can access it using URL without worrying about the underlying application Pod replicas serving the app. We also explored a way to publish apps to the internet using a single cluster-wide load balancer created using Ingress Controller.

In Chapter 3, Workload Objects - Pod, Deploy, StatefulSet, we briefly looked at Deployment and some of its features. In this chapter, we will further delve into Deployment, especially from the perspective of application scalability. We will also learn how to automatically scale apps based on key thresholds such as CPU and memory utilization.

Examples referred to in this chapter are run in the cluster that is set up using Kubeadm, as explained in Chapter 2, Installing Kubernetes.

Structure

In this chapter, we will cover the following topics:

Setup - CNI, Metrics Server

Deployment

Scalability

Auto-scaling/HPA - Horizontal Pod Autoscaler

Setup

As part of the Kubeadm setup, you will notice that the following components are installed:

API Server

ETCD

Kube-scheduler

Kube-proxy

CoreDNS

Kube Controller Manager

For Pod-to-Pod communication, we need Container Network Interface (CNI) implementation. Let us use the Calico CNI Adapter by running the following command:

curl

https://raw.githubusercontent.com/projectcalico/calico/master/manifests/calico.yaml -O && kubectl apply -f calico.yaml

Metrics Server

To monitor the CPU and memory utilization by Pods and Nodes, Metrics Server as an addon is available from the Kubernetes community. This add-on is necessary since it is used by the Horizontal Pod Autoscaler (HPA) controller.

Use the following command to install the metrics server:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.6.2/components.yaml

Since you are running the cluster locally, you need to update this server. Execute the following command, and you will see the associated YAML manifest gets opened in the default editor:

kubectl edit deploy -n kube-system metrics-server

Add the highlighted line (- --kubelet-insecure-tls), as shown in Figure 5.1, and save and exit the YAML manifest (using vi editor keys ESC :wq):

Figure 5.1: Metrics Server - Insecure Kubelet TLS

Verify if the metrics server Pod is running successfully using the following command:

vagrant@master-node:~$ kubectl get pods -n kube-system -l k8s-app=metrics-server

NAME                             READY   STATUS    RESTARTS   AGE

metrics-server-9c7c8cb7b-6ww9p   1/1     Running   0          4h2m

In a matter of a few minutes, the metrics server starts collecting CPU and memory metrics, and you can verify it using the following command. Refer to Figure 5.2 for reference:

Figure 5.2: Kube-system Pods Metrics

Deployment

One of the important benefits that Kubernetes provides is automatic resource (bin) packing. Based on the overall computing capacity available on the cluster nodes, Kubernetes decides the node suitable to run the given Pod. Runtime resources, mainly the CPU and memory (RAM), required by the container can be specified as part of the Pod template. The kube-scheduler considers these requirements while scheduling the Pod on a suitable node.

See the following example, the YAML manifest, which specifies the minimum CPU and memory requirements of the hello container using the requests block under the container resources section. It also includes the max CPU and memory that the application can have using the limits section. As per this YAML manifest, 10m (millicore) of CPU and 10Mi (MiB) of RAM are required as a minimum to run this container. This manifest also specifies the max CPU (50m) and memory (50Mi) this container can be allocated. The kube-scheduler ensures that the Pod declared in this YAML manifest is scheduled on a node where allocatable CPU and memory capacity is more than the CPU and memory requests that this Pod has. In case there is more than one container, then the sum of all the requests - CPU and memory - is taken into consideration by the kube-scheduler. Containers having no resource requirements specified are most likely to be evicted first by the kubelet when the node experiences resource shortages.

Figure 5.3: Deployment - hello with resources specification

This hello application can be deployed using the following command:

kubectl apply -f  https://raw.githubusercontent.com/brainupgrade-in/book-kubernetesadmin/main/chapter_05/deploy-hello.yaml

(NOTE: above command should be run on a single line)

Using imperative commands, the same can be achieved. Refer to the following commands:

kubectl create deploy hello --image brainupgrade/hello

kubectl set resources deploy hello --requests=cpu=10m,memory=10Mi --limits=cpu=50m,memory=50Mi

Tip: During the certification exam, imperative commands will save you a lot of time, so practice using imperative commands.

Verify that the Pod is running. In addition, check if resource utilization information is also available via metrics API. Refer to Figure 5.4 for reference:

Figure 5.4: Metrics - Pod and Nodes

To find out the node on which the Pod is running, run the following command:

kubectl get pods -owide

To know the allocated resources on the node where Pod is running, run the following command:

kubectl describe node worker-node01 | grep -A 999 “Non-terminated Pods”

The output will be similar to the following:

Figure 5.5: Allocated Resources - Node

Scalability

Now, let us test this hello web application using an easily available cloud load testing tool named artillery (https://github.com/artilleryio/artillery). The intent of load testing this app is to observe how the load is balanced (distributed) using replication (scalability). The quickest way to launch this tool is by using the following command:

kubectl run loadtest --image artilleryio/artillery --command tail -- “-f” “/dev/null”

Then, enter into this Pod by executing the following command. Refer to Figure 5.6 for reference:

kubectl exec -it loadtest -- sh

Figure 5.6: Artillery Launch

Before we start load testing the hello app, expose it using the following command:

kubectl expose deploy hello --port 80 --target-port 8080

To have a better visualization of the work we are doing, you can use the tmux utility (a terminal multiplexer that comes with Ubuntu by default).

Tip: This will also help speed up things a bit, especially during the certification exam. A few handful of tmux commands you need to remember and you will be benefited a lot! These are a few useful commands:

ctrl+b “ (to split the window horizontally)

ctrl+b % (to split the window vertically)

ctrl+b :set mouse on (to enable mouse)

ctrl+b PgUp/PgDown (to scroll up/down the window output)

ctrl+b+ (arrow keys to resize the window)

Refer to Figure 5.7 as a reference. On the right side, there are two small windows to monitor the Pods and Nodes’ resources using the Linux watch utility. For example, watch kubectl top pods. The left top window is for load testing the hello app, and the bottom window is for running kubectl commands.

Figure 5.7: Tmux View

On the artillery command line, we can fire this command that will create 100 virtual users (-c 100) and call the hello app 100 times (-n 100) per user.

/home/node/artillery # ./bin/run quick -c 100 -n 100 http://hello

The following screenshot was taken after the test when only one replica was running:

Figure 5.8: Load Test with 10000 Hits

The app responded with an average time of 314.2 milliseconds per request (for 99 percent of users) while only one replica was running.

By increasing the number of replicas, you can scale up the application and thus distribute the load. To scale up the app, say to 5 replicas, the following command can be used:

kubectl scale --replicas 5 deploy hello

Once replicas are up and running, you can fire the load test again using the artillery console. Test results are captured in Figure 5.9. As you can see, the average response time is reduced to 232.8 milliseconds, giving more than 20% performance improvement.

Figure 5.9: Load Test Results - 5 Replicas

Scaling the app not only improves the response time by distributing the load but also reduces the downtime if requests are too many to handle for the app.

Auto-scaling/HPA - Horizontal Pod Autoscaler

Manual scaling is used in an ad-hoc manner to balance the app, for example, when peak times are known. In a production setup, automatic scaling of replicas is often used.

HorizontalPodAutoscaler automatically updates the number of Pod replicas to match demand on the app. If the load on the app is higher, then the HPA creates more replicas, and when the load decreases, app replicas are reduced as well. HPA monitors the resources - CPU and/or memory - and takes action if the value of the resources crosses the threshold value.

Note: HPA gets the metrics information from the API provided by the metrics-server; hence, it is a must to have the metrics-server running in the cluster.

The quickest way to autoscale any app is to use the command similar to as given here:

kubectl autoscale deploy hello --min 1 --max 10 --cpu-percent 50

The preceding command creates an HPA specifying minimum replicas to be running as 1 (--min 1) and maximum replicas that can be created up to 10 (--max 10). The threshold parameter to be used is CPU percent and the value for the threshold is 50 (--cpu-percent 50). This percentage is of the request CPU field. If the average CPU utilization of all the active replicas is more than the threshold value, then the HPA controller immediately scales up the replicas; otherwise, replicas are scaled down. When it comes to scaling down replicas, the HPA controller takes some time (specified as stabilizationWindowSeconds with default 300) to remove replicas. This is done to provide cluster stability.

We can get more information about this HPA, including its current status, by executing the following command. Refer to Figure 5.10 for reference:

Figure 5.10: HPA - hello App

Now, let us generate a continuous load and observe how autoscaling works. To generate the load, we can use the following command:

/home/node/artillery # while true; done ./bin/run quick -c 100 -n 100 http://hello; done

Figure 5.11: Autoscale hello - 10 Replicas

From the preceding screenshot, you can see that 10 replicas are created by the HPA controller to manage the current load.

The moment you stop the load testing, you will notice that after 5 minutes, scaling down begins. That is the default scale-down behavior; however, you can customize it based on your requirements.

To add memory as one of the resources to monitor for autoscaling, you can edit the HPA manifest using the command kubectl edit hpa hello and add a resource block as seen in Figure 5.12 (lines 14 to 19). Once changes are applied, you can see the current status of hpa using the following command:

vagrant@master-node:~$ kubectl get hpa -owide

NAME    REFERENCE          TARGETS            MINPODS   MAXPODS   REPLICAS   AGE

hello   Deployment/hello   21%/50%, 10%/50%   1         10        1          3h20m

As you can see from the command output, CPU and memory are below the target threshold; hence, the number of replicas matches the minimum.

Figure 5.12: HPA - Memory Resource

Conclusion

In this chapter, we learned how to deploy the application by providing specifications on runtime resources needed by the application. This not only enables Kubernetes to do better placement of Pods on Nodes but also facilitates autoscaling. Addon component, metrics-server, monitors Pods and provides CPU and memory utilization metrics. Creation of HPA - HorizontalPodAutoscaler either through YAML manifest or using imperative command triggers HPA controller to monitor workloads and scale applications based on the threshold and current metrics values. If the current value, say CPU utilization, is more than the threshold CPU utilization value, then the HPA controller creates more replicas as long as the total replicas are less than the current ones and vice-versa.

The load on application is dynamic, and in this situation, the HPA controller tries to maintain cluster stability by immediately scaling up but scales down only when the load on the application is less than the threshold for a considerable period. However, this default behavior can be customized by specifying scale-up/down behavior in the HPA manifest.

In the next chapter, we will learn the deployment strategies supported by Kubernetes, such as rolling update and recreate. Using labels applied on the Pod and its associated Service, we can achieve blue/green (sometimes referred to as red/black as well) deployment strategy and canary deployment to some extent.

Points to Remember

The kube-scheduler uses resources spec defined for each container while scheduling the Pod.

Kubernetes uses HPA - HorizontalPodAutoscaler - controller to provide autoscaling of workloads based on the given metrics.

The Kubernetes addon, metrics-server, monitors and provides metrics (CPU and memory) information for Pods and Nodes. The HPA controller depends upon the metrics-server addon to obtain the CPU and memory utilization.

The default scaling behavior implemented by the HPA controller is immediate scaling up of replicas upon crossing the threshold and scaling down after a stabilization period of 300 seconds when the value of the measured metric goes below the threshold.

Multiple Choice Questions

Which one of the following is not provided as part of the vanilla Kubernetes setup?

API Server

ETCD

Kube-scheduler

Kube Controller Manager

CNI Adapter/plugin

Which one of the following resources cannot be specified as part of the container resources spec in the Pod manifest?

CPU

memory

hugepages-<size>

Network IO bandwidth

Which one of the following nodes would be used to deploy Pod having a resource request of 500m?

Node A (Capacity: 1 core, Allocated CPU requests: 300m)

Node B (Capacity: 2 cores, Allocated CPU requests: 1550m)

Node C (Capacity: 3 cores, Allocated CPU requests: 2510m)

Node D (Capacity: 4 cores, Allocated CPU requests: 3505m)

How many additional replicaSets would be created if CPU resource request is later added post-creation of Pod using the Deployment manifest?

0

1

2

3

To specify the max CPU that can be allotted to a Pod, which Service attribute should be set?

.spec.containers[].resources.requests.cpu

.spec.containers.resources.request.cpu

.spec.containers[].resources.limits.cpu

.spec.containers[].resources.limit.cpu

To specify the minimum CPU that should be allotted to a Pod, which Service attribute should be set?

.spec.containers[].resources.requests.cpu

.spec.containers.resources.request.cpu

.spec.containers[].resources.limits.cpu

.spec.containers[].resources.limit.cpu

Kubelet terminates the container if it consumes the max CPU allocated to it.

True

False

The HPA controller has definite scaleUp and scaleDown behavior that cannot be customized.

True

False

Answers

e

d

a

b

c

a

b

b

Questions

Can applications be autoscaled using custom metrics, for example, requests/second?

How does Kubernetes provide stable autoscaling behavior even though the nature of load on applications is dynamic in nature?

Keywords

HPA - Horizontal Pod Autoscaler

Resource requests and limits

Metrics-server

stabilizationWindow

Scalability

CHAPTER 6

Deployment Strategies – RollingUpdate, Recreate

Introduction

In the previous chapter, we learned how to scale stateless applications by specifying the resources (typically CPU and/or memory) required to run apps and then setting the threshold values for those resources so that HPA (Horizontal Pod Autoscaler) controller can perform the autoscaling. In addition, we also looked at the Metrics Server, a critical add-on component required by the HPA controller to function.

In this chapter, we will look at various deployment strategies that can be implemented using Kubernetes. These strategies are often employed by software engineering teams to release their software.

The examples referred to in this chapter are run in the multi-nodes cluster that is set up using the KIND method, giving cluster.yaml as an input configuration, as explained in Chapter 2, Installing Kubernetes.

Structure

In this chapter, we will cover the following topics:

Deployment Strategies

RollingUpdate

Recreate

Blue Green/Red Black

Canary

Deployment Strategies

The ways in which software should be rolled out are influenced by various factors, such as business requirements – the nature of change that is being rolled out, software development practices (waterfall, agile, and more), and the nature of the software – whether the software is large monolithic or cloud-native, and so on. Following are the strategies that are often adopted while releasing new changes:

RollingUpdate

Recreate

Blue Green (also known as Red Black)

Canary

Kubernetes Deployment spec supports RollingUpdate and Recreate strategies. Effective use of pod labels and selectors defined by the service one can implement the Blue Green deployment strategy. Canary Deployment can also be achieved to some extent using pod labels, deployment replicas, and service selectors.

RollingUpdate

When users expect high availability of applications and new changes are rolled out frequently, generally multiple times a day, then gradually rolling out new changes and simultaneously removing the old version may be a preferred approach. In Kubernetes, this can be done using RollingUpdate deployment strategy. This enables zero downtime for the application while rolling out new changes.

The Kubernetes deployment spec allows configuring options for the RollingUpdate strategy. Under the deployment template field spec, one can define the strategy type as RollingUpdate and a couple of inputs to alter its behavior. Please refer to the yellow highlighted box in Figure 6.1. Two inputs required to implement this strategy are as follows:

maxSurge: This indicates the maximum number of additional replicas that can be created while rolling out a new change. It is defined as % of the replicas (field: .spec.replicas) or even a number. In this case example, maxSurge number is 1 (25% of 4 replicas). So the total number of replicas that Kubernetes may create during rollout is 5 (replicas + maxSurge). If maxSurge is a fractional number, then the next highest real number is considered, for example, 3 replicas if maxSurge is 2.5 replicas.

maxUnavailable: To ensure the high availability of the application, one can specify how many replicas from the total replicas can be unavailable during rollout. In this example, maxUnavailable number is 1 (25% of 4 replicas). This means Kubernetes must ensure that the minimum number of available replicas should be 3 (replicas - maxUnavailable) during rollout. If maxUnavailable is a fractional number, then the lowest real number is considered.

Figure 6.1: RollingUpdate configuration

Kubernetes creates new replicas and simultaneously terminates old replicas while honoring the maxSurge and maxUnavailable constraints. Considering the preceding example, Kubnetes may have up to 5 replicas while ensuring a minimum number of available replicas to be 3.

Please refer to Figure 6.2, which captures updates (audit trail to the deployment resource app) made to the deployment. Kubernetes ensures that the available number of replicas is between 3 and 5, as seen in the yellow highlighted box, in order to achieve zero application downtime.

Figure 6.2: Rollout of new change

Recreate

Recreate strategy terminates all running replicas before launching new replicas having application upgrades. This strategy can be defined using the field .spec.strategy, as shown in Figure 6.3.

Figure 6.3: Recreate strategy

Once this strategy is configured in the deployment spec, then, at the time of new change rollout, Kubernetes terminates all the existing replicas before launching replicas for version 2.0 of the application, as seen in Figure 6.4.

Figure 6.4: Recreate strategy - implemented

Blue Green (aka Red Black)

In this strategy, two separate environments are kept running. One environment, also known as blue, runs the current version of the application, and the second environment (green) runs the newer version of the application. Once the new version of the app running in the green environment is tested, all new requests to the app are redirected to the green environment. In case, rollback is required, then requests are simply redirected back to the blue environment.

This strategy gives benefits such as releasing a new version of the app almost instantly, thus promoting application availability, and simplifying the rollback in case deployment fails, thus reducing deployment risk.

In Kubernetes, this is achieved by using pod labels, service selectors, and deploying two versions of the app using separate deployment specs. Consider the following example (see Figure 6.5), wherein the application named accounting is deployed using the deployment named accounting-blue, which represents version “1.0” of the application.

Figure 6.5: Blue Deployment - Accounting

Deployment named accounting-blue, as shown in Figure 6.5, is exposed via the service named accounting (see Figure 6.6).

Figure 6.6: Service – Accounting

The second deployment named accounting-green (see Figure 6.7) is also deployed, though not yet exposed via the accounting service.

Figure 6.7: Green Deployment - Accounting

Once deployed, we can test this application using the following commands:

kubectl run test --name brainupgrade/tshoot

kubectl exec -it test -- bash

curl accounting-blue

Please refer to the blue-marked region in Figure 6.8 for reference.

Figure 6.8: Testing the accounting application - blue version

While both versions are deployed, only the Blue version is serviced to the client via the accounting service object. When the app team decides to release the next version of the application, version 2.0, also referred to as accounting-green, the label on the service object is changed to version “2.0”, and immediately version 2.0 is exposed to the client via the accounting service. This switch from version 1.0 to 2.0 is almost instantaneous with no downtime, as the version 2.0 instance(s) is already running.

The labels on the service object can be changed either by directly modifying the service object (kubectl edit svc accounting) or by using an imperative command, as follows:

kubectl set selector svc accounting release=“2.0”

We can test the service to see if indeed version 2.0 of the app is served or not. Please refer to Figure 6.9 for reference.

Figure 6.9: Testing – version 2.0

In case, the app team decides to rollback to the previous version, that is, blue version, then service selectors are updated to rollback the release almost instantly. Please refer to Figure 6.10 for reference (red-marked regions).

Figure 6.10: Rollback to the blue version

Canary Deployment

This deployment technique is used to gradually roll out changes alongside production deployment. The intent is to test the new changes by releasing them to a subset of users before doing the full deployment. This is especially useful when new features are to be tested or critical changes are to be introduced. By releasing the changes to a subset of users, real-world testing of new changes can be performed, and if issues are found, then rollback can be done quickly without impacting the whole set of users negatively.

Using Kubernetes, a simple canary deployment strategy can be implemented by routing a small subset of user requests to the canary deployment where new changes are deployed. If the scope of the deployment outcome includes the user demographics, requests source, and more, then in addition to Kubernetes configuration, changes within the application and/or outside application, such as DNS routing, should be considered. The scope of this section is limited to using Kubernetes to implement simple canary deployment.

Please refer to Figure 6.11 for better understanding the process of deploying an e-commerce app and serviced through the cluster URL ecomm. As you can see from the example, version v1 (refer the label release: v1) has a total of 9 replicas.

Figure 6.11: Ecomm app - production deployment

The e-commerce development team intends to introduce an AI feature to a subset of users, which is 10% of the total user base. This can be achieved in Kubernetes by deploying the new AI feature using the following YAML snippet (see Figure 6.12).

Figure 6.12: Canary Deployment - AI Feature

Once canary deployment using the preceding YAML spec is done, then user requests to the Service URL ecomm will be routed to both deployments, that is, one having label release: v1 as well as canary having label release: feature-ai, while the latter will receive nearly 10% of user requests (replicas from canary deployment/total replicas).

Conclusion

In this chapter, we learned about four deployment strategies, namely, RollingUpdate, Recreate, Blue Green, and Canary. The first two strategies (RollingUpdate and Recreate) are directly supported through the deployment template itself, and the rest two strategies can be implemented using pod labels and selectors defined within deployment and service specs.

RollingUpdate is the most widely used and also the default strategy implemented by Kubernetes. This strategy provides the benefit of rolling out new changes without any application downtime. In case, new changes require the application to be fully taken down before rolling out the change, then Recreate strategy is used. The Blue Green deployment strategy is most effective when the change rollout (and rollback in case of failure) needs to happen quickly. This strategy involves current and new versions to be running in parallel. The Canary deployment strategy is used when rolling out new features/changes to a subset of users before releasing the new changes to all the users.

In the next chapter, we will learn different ways available for the application to persist its data. We will also discuss how applications can store the data locally on the node they are running or outside the node, such as using cloud storage.

Points to Remember

Zero downtime application rollout can be achieved using the RollingUpdate deployment strategy.

When Recreate strategy is specified, all the currently running replicas are terminated before new ones are launched.

In Blue Green deployment, a new version of the app is running in the Green environment which is a parallel environment. Once all the testing is done in this environment, user requests are redirected from the Blue environment to the Green one. In case any rollback is required, it can be done almost instantaneously.

The Canary deployment is used to try out new changes by making those available to a subset of users before releasing those to all the users. In this, the new app is running beside the current version of the app, and both the versions (current and canary) are serviced via a common service object. The subset of users is determined using the number of replicas on Canary as well as the current version of app deployment.

Multiple Choice Questions

To achieve incremental and iterative software release without downtime, which deployment strategy is best suited?

Recreate

RollingUpdate

Red Black

Blue Green

Canary

To test the experimental feature by allowing a subset of real users to test, which deployment strategy should be used?

Canary

RollingUpdate

Recreate

Blue Green

In which deployment strategy, two parallel running environments are created?

Recreate

RollingUpdate

Blue Green

Canary

In which deployment strategy, currently running replicas are fully terminated before launching new ones?

Canary

RollingUpdate

Blue Green

Recreate

Canary deployment can be specified using the deployment YAML template. Is it true or false?

True

False

How many replicas Kubernetes can create if maxSurge is set as 30% and replicas as 10?

3

12

7

13

How many replicas will always be available during RollingUpdate if the number of replicas is set as 4 and maxUnavailable as 25%?

1

2

3

4

Blue Green deployment strategy is configured using deployment YAML template. Is it true or false?

True

False

Answers

b

a

c

d

b

d

c

b

Questions

Which strategy requires running parallel environments?

Business team wants to introduce certain app features to users belonging to a particular geography only. Can this be achieved using Kubernetes?

Keywords

RollingUpdate

Recreate

Blue Green/Red Black

Canary

CHAPTER 7

Data Persistence - Local and Cloud

Introduction

In the previous chapter, we learned about the different strategies that can be used for deployment. RollingUpdate and Recreate are directly supported as part of the Deployment specification template. Blue Green and Canary deployment strategies can be implemented using pod labels and selectors within deployment and service. RollingUpdate is the default strategy implemented and provides zero downtime while rolling out new changes.

The lifecycle of data is tightly coupled with the lifecycle of storage on which it is kept. If data is persisted on the Node, then the data lifecycle is bound to the Node’s lifecycle, whereas if the Node is terminated, then the data on that Node is lost. If the lifecycle of data is required to be independent of Node’s lifecycle, then data should be persisted on external storage.

In this chapter, we will learn different ways to persist data. Different persistence options available to applications are pod, node, and cloud storage. Besides these, Kubernetes also provides ways to access configuration as well as secret information via file system mounting.

Examples referred to in this chapter are run in the multi-nodes cluster that is set up using the KIND method, giving cluster.yaml as an input configuration, as explained in Chapter 2, Installing Kubernetes. Examples associated with cloud storage, such as AWS EBS, may work when Kubernetes cluster is installed in AWS infrastructure.

Structure

In this chapter, we will cover the following topics:

Pod Storage (emptyDir)

Node Storage (hostPath)

Persistence Volume

Persistence Volume Claim

Persistent Volume Types

Access Modes

Storage Class

Cloud Storage - AWS EBS

Pod Storage (emptyDir)

This is a temporary storage solution provided to container(s) within a Pod. It is a simple and ephemeral storage option for sharing files among containers within the same pod. A temporary storage is created and allocated to the pod whenever it is scheduled onto a particular node and running on that node. The moment the pod is terminated, the associated storage is also removed.

Whenever this storage is first created, an empty directory is created. All the containers in the same pod can access it and read/write data into it during the pod’s lifetime. This makes emptyDir volume type an excellent choice for inter-container communication. This storage option can also be used by containers to store temporary data, which is available across container restarts. While this storage serves as a good scratch space for containers, it can also be used for caching files and sharing with other containers within the same pod.

The lifecycle of data stored in emptyDir volume is tightly coupled with that of the pod. Whenever a pod is deleted, evicted, or rescheduled onto a different node, this storage will be lost. Please refer to Figure 7.1, in which an image-processor container uses emptyDir for its temporary/scratch work at the mountPath /scratch inside the container.

Figure 7.1: emptyDir - image processor

Please refer to Figure 7.2, in which the pod ml-training-pod uses emptyDir storage as shared storage to coordinate the work between containers data-preparation and model-training.

Figure 7.2: emptyDir - shared storage

The medium used for storage can be changed to a RAM-backed file system by setting the field emptyDir.medium: “Memory”. Otherwise, emptyDir volumes are stored on whatever medium that backs the node, such as SSD by default. To specify a size limit, you can set the field emptyDir.sizeLimit by enabling the feature gate SizeMemoryBackedVolumes.

Node Storage (hostPath)

The hostPath volume type in Kubernetes allows the mounting of files, directories, sockets, characters, or block devices into the pod’s containers. This is useful for accessing and sharing the data directly from the host machine on which the pod is running.

In Figure 7.3, we can see that a web-server container in the nginx-cache pod uses the host filesystem to build and maintain the cache needed for its work. If the directory (/var/nginx-cache) does not exist, it will be created. Otherwise, the existing directory, including its content, will be mounted inside the web-server container.

Figure 7.3: hostPath - nginx cache

If any other replica of this nginx-cache gets scheduled and running on this same node (/host), then both the replicas can make use of this hostPath as shared storage.

NOTE:

HostPaths can expose privileged system credentials or APIs, which can be used for container escape, potentially leading to attacks on the cluster or the network.

Pods with identical information may behave differently on different nodes based on the different filesystem on the nodes.

Files or directories created are only writable by root.

Persistent Volume

It is often desirable to abstract the details of how storage is provided from how it is consumed. To achieve this, Kubernetes provides two APIs: PersistentVolume and PersistentVolumeClaim. Consumers or users use claims (PersistentVolumeClaim) to request storage (PersistentVolume).

A PersistentVolume (PV) can be provisioned manually by an Administrator, a process also known as static provisioning. To dynamically provision PersistentVolumes, StorageClasses can be used. This PV captures the details of storage implementation, whether it is NFS, iSCSI, or a storage provided by any cloud provider.

PersistentVolumeClaim

A PersistentVolumeClaim is a request for storage by a Pod. Pods can use claims to request storage of specific size and access modes, indicating how storage can be mounted, such as ReadWriteOnce, ReadWriteMay, or ReadOnlyMany. To cater to various users’ demands regarding storage performance, volume expansion, and other requirements, Administrators use StorageClasses containing storage implementation details, thus abstracting the same from the users while fulfilling their requirements.

Please refer to Figure 7.4 for reference. The pod nginx-cache uses the claim named claim-cache-vol to refer to the persistent storage.

Figure 7.4: Nginx app using PVC

Actual storage details are encapsulated in the PersistentVolume object named vol-nginx-cache (Figure 7.5). Most clusters define a default storage class, and to avoid creating storage dynamically, you should set storageClassName as blank (line 11 in PVC - Figure 7.4).

Figure 7.5: PV for Nginx app

There is a one-to-one binding between PVC and PV. Therefore, if you want to ensure that a particular PV is used by a PVC, then you must set appropriate fields in both the PV (claim reference, that is, PVC) and the PVC (name of the PV).

PersistentVolume Types

PersistentVolume types are implemented as plugins. As of this writing, the following plugins are supported by Kubernetes:

hostPath: storage on a single node only

csi: Container Storage Interface - CSI

fc: fiber channel storage

iscsi: SCSI over IP storage

local: local storage devices mounted on nodes

nfs: network file system storage

Access Modes

A PersistentVolume can be mounted on a host using any of the following access modes. Details on which access mode(s) is supported are typically documented by the storage resource providers.

ReadWriteOnce (RWO): The volume can be accessed as read-write by a single node.

ReadOnlyMany (ROX): The volume can be mounted as read-only by many nodes.

ReadWriteMany (RWX): The volume can be mounted as read-write by many nodes.

ReadWriteOncePod (RWOP): The volume can be mounted as read-write by a single pod.

Storage Class

Kubernetes provides integration with various storage classes offered by storage providers. Different classes of storage are described using StorageClass. Kubernetes administrators define StorageClasses to fulfill on-demand storage requirements by Pods. The default storage class will be used to create a PV for any PVC that does not specify a storageClassName as part of its spec. If any PVC refers to any specific storageClassName, then that storage class is used to provision the storage; otherwise, the Kubernetes Administrator is expected to provide the storage manually.

Please refer to Figure 7.6, in which the AWS EBS provisioner is specified. AWS EBS storage plugin is also installed (using instructions are found at https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/install.md), which is then used to fulfill any storage requirements that refer to this storage class named gp2.

Figure 7.6: Storage Class - AWS EBS

Figure 7.6 shows that the storage class is also set as default using the annotation (line #6). This StorageClass uses gp2 as the file system (line #9). The storage created using this class may be retained after the termination of PV (line #10). In case more volume is required than earlier requested, then by modifying the created PV, more storage can be obtained. This is possible only when the field allowVolumeExpansion is set to true (line #11). Whenever any PVC is created, the PV is immediately created and bound to the PVC (line #14). This binding mode may leave the pod unscheduled if topology constraint doesn’t allow scheduling of the pod on the node where the volume is bound. In such cases, it is best to set the binding mode as WaitForFirstConsumer.

Cloud Storage - AWS EBS

Please refer to the following example, in which the pod (nginx-cache-ebs) uses a PVC (claim-cache-ebs) to obtain storage offered by the cloud service provider – AWS, as shown in the Figure 7.7. The StorageClass named gp2 is used to dynamically provision the storage based on the gp2 file system. If this example is run on Kubernetes deployed on AWS infrastructure, you would see a PV getting created by the AWS EBS plugin based on the StorageClass (gp2).

Figure 7.7: Pod referring EBS PVC

Figure 7.8: Storage Class - AWS EBS

Conclusion

In this chapter, we learned about the types of volumes supported by Kubernetes. Containers can use Pod storage, which survives container crashes. The Pod storage can also be used to facilitate inter-container communication within the same pod. If pods require node storage to effectively collaborate, when running on the same node, then hostPath is used. Cloud storage, such as AWS EBS, can be used when the availability of storage is expected to survive both pods and node crashes.

To abstract how volumes are provisioned from its usage, PVCs are used. To create PV dynamically, StorageClasses are used to provision storage whenever PVCs are created specifying a storageClassName. In case, PVC does not specify a storageClassName, then the default StorageClass, if any, is used to provision the storage.

In the next chapter, we will look at the StatefulSet API object and how it can be used to launch clustered databases like Redis.

Points to Remember

An Empty directory is created if it does not exist when emptyDir Pod storage is defined.

The volume hostPath refers to the path on the node where the pod is currently running. It is suitable for single-node use only.

PVC (PersistentVolumeClaim) is used to abstract the PV (PersistentVolume) creation details from its usage within a pod.

Pods use PVCs to claim PV - PersistentVolume. There is a one-to-one binding between PVC and PV.

Pods cannot refer to PVCs outside their own namespace.

StorageClasses are used to create different classes of storage. Volume plugins associated with the StorageClasses provision the storage requested.

If PVC does not specify storageClassName to be used, then default StorageClass, if any, will be used to provision storage for that PVC.

Multiple Choice Questions

What is the key characteristic of emptyDir?

It retains data even after Pod is terminated

It is an ephemeral volume that gets created when Pod is scheduled on a node and is removed when Pod is deleted

It is shared across multiple pods

The lifecycle of emptyDir is independent of Node’s lifecycle where Pod is running

Which scenario is most appropriate for the use of emptyDir?

To store critical data

To share data across multiple pods

To share data across containers within the same pod

To provide a durable storage solution

Which one of the following is true in the case of hostPath volume?

It mounts a file/directory from the host file system into a container in the pod

It provides highly available and durable storage

It is automatically provisioned when a pod is created

It allows sharing of data among pods running on different nodes

In which scenario you might consider using hostPath volume?

When you need to share data among multiple pods running on the same host

To store sensitive user data securely

When you require persistent storage with automatic replication

To store configuration files within a pod

What is the significance of persistentVolumeReclaimPolicy in PV definition?

It determines the access mode in PV

It specifies storage capacity

It defines how the PV’s storage should be reclaimed once its associated PVC is terminated

It designates a namespace where PV will be available

What is the purpose of PV - PersistentVolume?

To store configuration files that can be shared across pods

To provide high-speed storage

To manage application deployments in Kubernetes

To define storage resources that can be accessed via PVC, hence abstracting the details of storage implementation from how it is used

What is the primary purpose of StorageClass?

To define the storage capacity of PV – PersistentVolume

To specify the access mode for PVC – PersistentVolumeClaim

To provide a way to dynamically provision and manage PV

To back up the storage before deletion

StorageClass automates the process of creating and managing PVs based on predefined rules.

True

False

Answers

b

c

a

a

c

d

c

a

Questions

What is the key difference between emptyDir and PersistentVolume?

What is the use of StorageClasses? How are they related to PVCs?

Keywords

emptyDir

hostPath

PersistentVolume

PersistentVolumeClaim

StorageClass

volumeBindingMode

accessModes

CHAPTER 8

Deploy and Scale - StatefulSet

Introduction

In the previous chapter, we focused on data persistence through various Kubernetes storage options such as pod storage with emptyDir, node storage using hostPath, and cloud storage, notably AWS EBS. It introduces PersistentVolume (PV) and PersistentVolumeClaim (PVC) as methods to abstract storage details from users, with PVCs requesting specific storage features and PVs fulfilling those requests, potentially through dynamic provisioning using StorageClasses. The chapter also discusses access modes for PVs and the concept of storage classes, which define different storage types that can be dynamically provisioned based on class specifications. The chapter emphasizes the importance of understanding the lifecycle of data in relation to the lifecycle of the node or pod it resides on, with persistent cloud storage offering data longevity beyond pod or node terminations.

In this chapter, we will learn StatefulSet, a high-level API resource that effectively handles the deployment and scaling of stateful applications by assigning each pod an exclusive and stable identifier. It also ensures that the individual identity of each pod persists throughout its lifecycle. It is designed with stateful workloads in mind, such as databases, to provide each pod with its own persistent storage, ensuring data is preserved across pod terminations and restarts. It follows a strict order and methodology for updates and scaling, which is essential for stateful applications that depend on a specific initialization and teardown sequence. StatefulSets facilitate controlled rolling updates, allowing for seamless upgrades and configuration changes with minimal service disruption.

Examples referred to in this chapter are run in the multi-nodes cluster that is set up using the KIND method, giving cluster.yaml as an input configuration, as explained in Chapter 2, Installing Kubernetes.

Structure

In this chapter, we will cover the following topics:

Overview

Key Features

Stable Network Identifiers

Persistent Storage

Ordered Scaling

Automated Rolling Updates

Parallel Pod Management

Use Case - Redis Database

Scaling

Pod Management Policies

Rolling Updates

Partitioned Rolling Updates

Limitations

Overview

Not all applications are ephemeral and some do require persistent data storage to work with. StatefulSets ensure that when dealing with stateful applications, like databases, each pod replica is distinguishable from its other replicas, not just by its temporary existence but also by a consistent identifier across rescheduling or restarts. The persistent storage attached to each pod remains with the pod even after restarts or rescheduling on another node. It’s a stark contrast to Deployments that treat each replica as an interchangeable cog in a machine.

With StatefulSets, the pods are created sequentially and also terminated in reverse order, ensuring a predictable deployment and scaling process. This creates an environment where order matters, where the first pod must be ready before the second begins its life cycle, thus paving the way for graceful deployments and scale-ins. Moreover, this approach benefits applications that rely on a quorum or require orderly bootstrapping.

Each pod gets a stable hostname derived from the name of the StatefulSet and its ordinal index, which persists across rescheduling. Coupled with Headless Services, which provide a consistent network identity, StatefulSets make it possible for applications to discover their peers within the cluster and communicate with ease, enabling patterns of distributed systems.

Key Features

Following are the key features of StatefulSet:

Stable, Unique Network Identifiers

Each pod in a StatefulSet is assigned a unique ordinal index, which is used to construct its stable hostname. This ensures that each pod can be easily identified and communicated with, making them particularly suitable for clustered applications and distributed systems.

Stable, Persistent Storage

The persistent storage volume associated with StatefulSet remains associated with pod rescheduling and restarts. This allows the data to survive pod failures and disruptions, which is essential for stateful applications that require persistent storage such as databases.

Ordered, Graceful Deployment and Scaling

Pods in a StatefulSet are created and deleted in a predictable order. This order is based on the ordinal indexes of the pods, with higher indexes being created after lower ones and deleted before them. This is crucial for systems where the startup and shutdown order of pods matters.

Ordered, Automated Rolling Updates

When StatefulSet is updated, the updates are rolled out in a controlled fashion. This means that only one pod is updated at a time, in order of their ordinal. This approach is important for ensuring that updates do not disrupt the application’s availability or consistency.

Parallel Pod Management

Kubernetes 1.7 and later versions include the option for parallel pod management of StatefulSets, where pods are created and deleted simultaneously. This can speed up the scaling process, although it’s typically used in situations where the order of pod startup and shutdown is not critical.

Use Case - Redis Database

Let us explore the following example, where a clustered redis database use case is implemented using StatefulSet:

Figure 8.1: Cluster of redis database

In the preceding YAML:

Line #4: StatefulSet object is named as redis.

Line #6 : replicas: 3 means the redis app should have 3 instances.

Line #15: initContainers section contains one init container that does the initial configuration required for redis instances before making them ready for use. It uses the arguments (lines # 20 to 39, kept folded in the figure for clarity) to do the same.

Line #17 and 47: The image used for the init container and main container is redis:6.2.3-alpine.

Line #41: Configuration required to launch the redis database refers to this location /etc/redis.

Lines #61 to 63: Key configuration required by redis is kept in a separate Kubernetes Config object named config defined under volumes block.

Lines #65 to 71: The volume required for each of the replicas is specified under the volumeClaimTemplates block.

To launch this clustered redis database app, please run the following commands:

Create Kubernetes Config object first:

kubectl apply -f \

https://raw.githubusercontent.com/brainupgrade-in/book-kubernetesadmin/main/chapter_08/01a-config.yaml

Now launch the redis app:

kubectl apply -f \

https://raw.githubusercontent.com/brainupgrade-in/book-kubernetesadmin/main/chapter_08/01c-redis.yaml

You can create a headless service that can be used by the clients to access this app. This step is optional.

kubectl apply -f \

https://raw.githubusercontent.com/brainupgrade-in/book-kubernetesadmin/main/chapter_08/01b-service.yaml

Figure 8.2 shows that all replicas are in a running state. You can also see the PVCs (PersistentVolumeClaims) and associated volumes for these 3 replicas:

Figure 8.2: Redis replicas and volumes associated

Replica names consist of the name of the StatefulSet, which is used as a prefix and the ordinal index as a suffix, for example, redis-0. This is also the hostname for the replica. This hostname is a stable and unique network identifier and is used while accessing the replicas.

Similarly, their associated PVCs contain the name as defined in the volumeClaimTemplate followed by the replica name. To make things even clearer, see the following in case of a redis-0 replica:

redis-0 replica refers to data-redis-0 PVC, which in turn refers to the volume pvc-03293442-b250-46ff-b263-7fd45f506e61

Kubernetes guarantees that the same volume will be provided to this replica across pod rescheduling, for example, if pod replica-0 crashes and gets rescheduled on another node, say node-2, Kubernetes will make the same volume available on node-2 so that it can be accessed by the replica-0.

Scaling

For this Redis app, we created three replicas. Let us scale up the number of replicas to 5 and observe the scaling-up process.

Use the following command to scale up the replicas to 5:

kubectl scale --replicas 5 sts redis

In another window, you can also watch the audit trail of pod events to understand how replicas are created one after another in an orderly manner. This is typical of StatefulSet. Use the following command to watch the pod events audit trail:

kubectl get pods -w

Figure 8.3 captures the scale-up process:

Figure 8.3: Scaling up and pod events audit trail

As you can see from Figure 8.3, three replicas existed before the scale-up operation. Hence, redis-3 (the fourth replica) was created first, and only after replica-3 got into Running status, the fifth replica redis-4 was launched.

Let us experiment with scaling down and observe the behavior. To scale down replicas, use the following command:

kubectl scale --replicas 1 sts redis

To observe the ordered termination of replicas, use the following command:

kubectl get pods -w

Figure 8.4: Scaling down and ordered termination of replicas

From Figure 8.4, you may have observed that while replicas are terminated in order, from the higher ordinal index to the lower, and the current number of replicas is 1, the number of PVCs and associated volumes are still 5. The reason behind keeping those volumes is to ensure that replicas get the volumes they had before scaling down. Clustered database applications, like redis, expect to have the same data when the replica is relaunched (scaling up in this case).

Let us now scale up replicas to 5 and verify if redis-4 gets a reference to data-redis-4 PVC:

kubectl scale --replicas 5 sts redis

Run the following command to verify:

kubectl describe pod redis-4 | grep -A3 -i volumes

Figure 8.5: Scale up and verify replica-4 associated with data-redis-4 PVC

Pod Management Policies

StatefulSet provides uniqueness and identity guarantee besides ordered rollout and scaling. However, one can choose to rapidly scale up/down replicas by specifying parallel pod management.

To enable parallel pod management, the spec field .spec.podManagementPolicy should be set as Parallel.

Rolling Updates

Automated rolling updates are supported by StatefulSet. During rolling updates, the replica having the highest ordinal number will get updated first and then next lower, and so on. This is the default update strategy for StatefulSet. For example, redis-4 will be updated first, then redis-3, then redis-2, and so on.

To roll out updates to this redis app, use the following command:

kubectl set image sts redis redis=redis:6.2.4-alpine

You can observe the automated rolling update using the following command:

kubectl get pods -w

Referring to Figure 8.6, the redis-4 replica is first terminated, and the updated version of redis-4 is rolled out first. This same sequence is then followed for subsequent replicas in this order: replica-3 replica-2 replica-1 replica-0

Figure 8.6: Automated rolling updates - redis app

Partitioned Rolling Updates

In case you want to roll out changes to replicas having a certain ordinal number and above, then you can set that ordinal number by updating the StatefulSet object. The field to update is .spec.updateStrategy.rollingUpdate.partition.

Limitations

While working on StatefulSet, the following limitations should be kept in mind:

The storage requested by the pod should either be auto-provisioned using the appropriate storage class or pre-provisioned.

To obtain the network identity of the pods, the creation of Headless Service (https://raw.githubusercontent.com/brainupgrade-in/book-kubernetesadmin/main/chapter_08/01b-service.yaml) is required.

There is no guarantee that the replicas will be deleted once StatefulSet is deleted.

Manual repair is required if the default updateStrategy fails to launch any replica.

The storage associated with the replicas will not be deleted during the scaling-down operation. If it is desired to delete the associated storage, then the field .spec.persistentVolumeClaimRetentionPolicy.whenScaled should be set to Delete.

When StatefulSet is deleted, only the associated replicas are deleted and not the volumes created. If it is desired to delete the volumes, then .spec.persistentVolumeClaimRetentionPolicy.whenDeleted should be set to Delete.

NOTE: Persistent Volume Claim Retention Policy will be taken into consideration only when the feature gate StatefulSetAutoDeletePVC should be enabled on the API Server.

Conclusion

In this chapter, we learned that StatefulSets are a powerful feature in Kubernetes designed for managing stateful applications with unique identity, persistent storage, and ordered, graceful deployment and scaling. They ensure consistent and predictable handling of pods, which is essential for applications like databases that demand stable networking and data preservation. Through this mechanism, Kubernetes facilitates not only high availability but also reliability and data integrity across pod lifecycles. Despite some limitations and the need for careful management, StatefulSets are integral for deploying complex workloads, providing the necessary tools to maintain service continuity and statefulness in a cloud-native environment.

In the next chapter, we will look at the nuances of preparing applications for production deployment in Kubernetes, focusing on the importance of environment variables for app configuration. For non-sensitive data, we explore how ConfigMaps streamline the deployment process, whereas for confidential information, we turn to Secrets to ensure security. The chapter further emphasizes the resilience of production apps, facilitated by Kubernetes’ probe APIs — livenessProbe, startupProbe, and readinessProbe — which collectively enhance the availability and self-healing capabilities of applications. Through these probes, Kubernetes offers a robust system to check the health and readiness of applications, enabling smooth operations and minimal downtime in production environments.

Points to Remember

StatefulSet for Stateful Workloads: Leverage StatefulSet for applications like databases that need persistent storage and a unique identity, ensuring continuity even when pods are redeployed.

Sequential Scaling: StatefulSets manage pods’ creation and deletion sequentially, according to their ordinal index, which is crucial for maintaining the intended deployment order of stateful applications.

Consistent Pod Identity: With StatefulSets, each pod has a stable network ID that stays the same even when the pod instance is re-created, making network communication more reliable.

Headless Services for Networking: To facilitate stable networking within a StatefulSet, use a headless service. This allows each pod to be individually addressable without relying on load balancing.

Dedicated Persistent Storage: By defining volumeClaimTemplates, StatefulSets ensures each pod has its own PersistentVolumeClaim, maintaining a one-to-one mapping between the volume and the pod for data persistence.

Controlled Rollout of Updates: When updating a StatefulSet, the changes are applied one pod at a time in reverse ordinal order, reducing the risk of service interruptions during the update process.

Flexible Pod Management: For workloads where ordering isn’t critical, consider enabling parallel pod management to expedite scaling operations.

Pod Management Policies: Utilize the .spec.podManagementPolicy to define how pods within a StatefulSet are managed. Choose between “OrderedReady” for sequential management or “Parallel” for faster scaling.

Intervention for Update Failures: Be ready to step in manually if an automated update to a StatefulSet doesn’t go as planned, to troubleshoot and correct any issues.

Handling PVCs on Deletion: By default, PVCs persist after a StatefulSet is deleted. Manage the retention or deletion of these PVCs carefully, using the persistentVolumeClaimRetentionPolicy in Kubernetes 1.16 and later to avoid unintended data loss.

Multiple Choice Questions

What is the primary purpose of using a StatefulSet in Kubernetes?

To manage stateless applications

To manage stateful applications that require persistent storage and unique identities

To automate the deployment of services

To manage the network policies in a cluster

How do StatefulSets ensure that pods are created and deleted in a specific sequence?

By assigning each pod a random identifier

By utilizing a LoadBalancer

By using an ordinal index for each pod

By labeling each pod with a timestamp

What kind of service is typically used to provide stable networking for StatefulSet pods?

ClusterIP service

NodePort service

Headless service

ExternalName service

What Kubernetes object is used by StatefulSets to ensure that each pod has its own PersistentVolumeClaim?

PersistentVolume

volumeMount

volumeClaimTemplates

StorageClass

What update strategy allows for the controlled rollout of updates in a StatefulSet?

Recreate

RollingUpdate

OnDelete

Canary deployment

Which pod management policy can speed up the deployment and scaling of StatefulSets when the order of deployment is not crucial?

OrderedReady

Sequential

Parallel

Randomized

If a StatefulSet update fails, what is the recommended course of action?

Immediately delete the StatefulSet

Scale up the StatefulSet

Manual intervention and troubleshooting

Automate the repair with a CronJob

What happens to PersistentVolumeClaims (PVCs) when a StatefulSet is deleted by default in Kubernetes versions prior to 1.16?

They are automatically deleted

They remain until manually deleted

They are archived for future use

They become available for other pods

Answers

b

c

c

c

b

c

c

b

Questions

What specific characteristics of an application make it more suitable for a StatefulSet rather than a Deployment in Kubernetes, and how does the choice between the two affect application resilience and scalability?

How do the persistence requirements of an application influence the configuration of volumeClaimTemplates within a StatefulSet, and what considerations should be made regarding storage classes and access modes?

When configuring the update strategies for a StatefulSet, what are the trade-offs between using RollingUpdate and OnDelete, and how can these strategies impact the application’s availability during updates?

In what scenarios would setting the pod management policy to Parallel significantly improve the deployment time, and are there situations where the OrderedReady policy would be more advantageous despite its slower rollout?

Given the fact that PersistentVolumeClaims (PVCs) are not deleted when a StatefulSet is deleted by default, what steps should be taken to manage these resources, and how might this affect disaster recovery and data retention policies?

Keywords

StatefulSet

PersistentVolume

PersistentVolumeClaim

VolumeClaimTemplates

Headless Service

Pod Identity

Rolling Update Strategy

Pod Management Policies

Scaling

CHAPTER 9

Configure Apps for Production Deployment

Introduction

In the previous chapter, we explored the deployment and scaling of StatefulSet applications, a pivotal step for managing stateful workloads in Kubernetes with grace and precision. We learned about the unique aspects of StatefulSets, such as stable networking and persistent storage, which are critical for applications that require stable identities or ordered deployment and scaling.

As we transition to Chapter 9, we’re set to build upon that foundation by shifting our focus to the final and critical phase of application deployment: configuration for production environments. Here, we will unravel the best practices for utilizing environment variables to configure applications, employing ConfigMap objects for non-sensitive data, and leveraging Secret objects for confidential information. Furthermore, we will delve into Kubernetes’ self-healing mechanisms via probe APIs — liveness, startup, and readiness probes — that monitor and ensure the health and availability of our applications in production. This chapter promises to be a comprehensive guide to making our applications resilient and production-ready.

Examples referred to in this chapter are run in the multi-nodes cluster that is set up using the KIND method, with cluster.yaml provided as an input configuration, as explained in Chapter 2, Installing Kubernetes.

Structure

In this chapter, we will cover the following topics:

Overview

ConfigMap

Container env

Volume mounts

Command Line Argument

Secrets

Container env

Volume mounts

Command Line Argument

Container Probes

Startup

Readiness

Liveness

Probe Mechanisms

Overview

Packaged applications (nothing but the containers) are generally agnostic of the environment, allowing for a smooth transition from development to production by externalizing configuration particulars. ConfigMap and Secret objects, along with various probes, are fundamental for managing production applications in Kubernetes due to their roles in configuration management. While non-sensitive data is handled using ConfigMap, Secrets are critical for handling sensitive data, ensuring that passwords, tokens, and keys are stored and accessed securely - an absolute necessity for maintaining the integrity and security of production systems. Meanwhile, probes like livenessProbe, readinessProbe, and startupProbe are integral for maintaining application health and availability. They enable Kubernetes to make informed decisions about when to restart a container (self-healing), whether a container is ready to handle requests, and if an application has started correctly. This orchestration ensures that the applications remain resilient, managing failures gracefully, and thus sustaining the reliability of the production environment.

ConfigMap

To keep the configuration data separate from the application, ConfigMap is used. ConfigMap is an API object that holds data in key-value pairs. Applications can refer to the data in ConfigMaps as environment variables, command line arguments, or configuration files by mounting ConfigMap as volume inside the pod. This enables applications to be portable since environment configurations are decoupled from the packaged application.

NOTE

ConfigMap can store data up to 1 MiB.

Container env

Please refer to the example in Figure 9.1 of a ConfigMap object named weather-config. It contains three environment variables under the data block:

Figure 9.1: ConfigMap - weather-config

The weather application makes use of these environments by referring to this weather-config ConfigMap object and corresponding keys defined in the ConfigMap. Please refer to Figure 9.2 for more details. For the environment named ENVIRONMENT, a value is obtained using configMapKeyRef that takes the name of the ConfigMap weather-config and the key ENVIRONMENT as defined in the weather-config. Similarly, values for environment variables API_URL_WEATHER and API_URL_GEOCODING are obtained.

To run this example, execute the following command to create ConfigMap object:

kubectl apply -f \

https://raw.githubusercontent.com/brainupgrade-in/book-kubernetesadmin/main/chapter_09/01a-weather-config.yaml

Run the following command to launch the weather application:

kubectl apply -f \

https://raw.githubusercontent.com/brainupgrade-in/book-kubernetesadmin/main/chapter_09/01b-weather-app.yaml

Expose this application as a service by running the following command:

kubectl apply -f \

https://raw.githubusercontent.com/brainupgrade-in/book-kubernetesadmin/main/chapter_09/01c-weather-svc.yaml

To test the application, you can launch any pod that has curl utility. See the following:

kubectl run test --image brainupgrade/tshoot

kubectl exec -it test -- bash

Figure 9.2: Weather App with configMapKeyRef

This weather application outputs all environment values when its API endpoint /env is accessed using curl (curl weather/env). Please refer to Figure 9.3, and you will see the three environment variables (highlighted in red colored rectangle) apart from other system set values:

Figure 9.3: Environment variables for weather app

The weather app uses the API URLs set via environment config and provides temperature when the place name is provided via the endpoint /place/<name>. Please refer to Figure 9.4:

Figure 9.4: Weather app - temperature API

To load all environment variables defined in the ConfigMap, you can use envFrom. Please refer to Figure 9.5:

Figure 9.5: Load all env variables

Volume mount

Some applications use properties files to read environment data. Kubernetes allows grouping of related properties in ConfigMap and mounting them as files using volumes. For example, refer to the following example in which mail configuration properties are grouped together under the index mail.config in the ConfigMap weather-config (Figure 9.6). These properties are then mounted using volume named config inside the container at the path /etc/app/application.properties (Figure 9.7):

Figure 9.6: Config data using properties file

Figure 9.7: Referring properties file using volumes

You can use the commands shown in Figure 9.8 to verify whether configuration data is grouped together in the file application.properties and mounted at the path /etc/app:

Figure 9.8: Mount application properties

Command Line Arguments

Environment variables obtained using configMapKeyRef can also be supplied as command line arguments to the container. Please refer to Figure 9.9, which contains only the relevant portion of the YAML for brevity:

Figure 9.9: Command line arguments

Any updates in the ConfigMap object will be made available whenever the app is restarted (generally using the rollout restart command) if the ConfigMap data is consumed using env. If it is consumed via the filesystem, then no restart is required, as Kubelet generally refreshes filesystems quickly.

Secret

In Kubernetes, a Secret object is a way to store and manage sensitive data, such as passwords, OAuth tokens, and SSH keys. It’s essential for security purposes as it keeps this information out of your application code and prevents the data from being exposed in your deployment scripts or in logs. Using Secrets is a best practice for confidential data management, allowing you to maintain the confidentiality and integrity of your application without compromising its functionality or safety.

Let’s consider a scenario where we need to securely pass database connection details to the weather app. First, create a Secret object to store the database details. Please refer to Figure 9.10 for an illustration. The Secret data key values are encoded using base64 encoding. You can use the base64 utility to create encoded values, or alternatively, use the following command (single line) to obtain the corresponding YAML:

kubectl create secret generic weather-secret --from-literal=db_username=weather --from-literal=db_password=pwd12 --from-literal=db_root_password=secr123 --from-literal=db_name=weatherdb --from-literal=db_host=weatherdb --dry-run=client -oyaml

In this YAML (Figure 9.10), we define a Secret named weather-secret. The database connection fields, such as db_host, db_name, db_password, db_root_password, and db_username hold base64-encoded values. The mentioned command outputs YAML only, since --dry-run=client -oyaml options are added to the command. To create the weather-secret Secret object, run the following command:

kubectl apply -f https://raw.githubusercontent.com/brainupgrade-in/book-kubernetesadmin/main/chapter_09/04a-weather-secret.yaml

Figure 9.10: Secret data - weather app

The database application, named weatherdb, is a StatefulSet Kubernetes object based on the bitnami/mongodb container image and is supplied with these secret data. Please refer to the YAML snippet in Figure 9.11, wherein env values are obtained using the secretKeyRef named weather-secret and appropriate keys as defined in the Secret object.

To create this weatherdb database, run the following command:

kubectl apply -f https://raw.githubusercontent.com/brainupgrade-in/book-kubernetesadmin/main/chapter_09/04b-weather-db.yaml

Once the database is up and running, expose the database via a Service URL so that the weather app can use it:

kubectl apply -f \

https://raw.githubusercontent.com/brainupgrade-in/book-kubernetesadmin/main/chapter_09/04c-weather-db-svc.yaml

Now, launch the weather frontend app (Deployment Kubernetes object) as well using the following command. Database connection details to the weather are supplied as environment variables using secretKeyRef similar to the weatherdb:

kubectl apply -f \

https://raw.githubusercontent.com/brainupgrade-in/book-kubernetesadmin/main/chapter_09/04d-weather-app.yaml

Figure 9.11: Weather database

Expose this weather frontend app using the following command:

kubectl apply -f https://raw.githubusercontent.com/brainupgrade-in/book-kubernetesadmin/main/chapter_09/01c-weather-svc.yaml

Since the weatherdb and weather frontend are now working, we can do a quick test to verify if both of these application components using Secret weather-secret work well. Launch a test pod using the following command if not done already:

kubectl run test --image brainupgrade/tshoot

To fetch the temperature records from the database via the weather frontend app, run the following commands:

kubectl exec -it test -- bash

curl weather/history/temperature

To obtain temperature data for any place, run the following command:

curl weather/place/Bangalore

This command will hit the API endpoint /place/<place name> of the weather frontend app. In response to this API request, the app will fetch the temperature of the requested place besides saving the response in the backend MongoDB database. To obtain all the saved temperature data, the API endpoint /history/temperature can be used again. Please refer to Figure 9.12 for details:

Figure 9.12: Temperature records

Secret data can be supplied to pods via env as well as via filesystem by mounting volumes.

Any updates in the Secret object will be made available whenever the app is restarted (generally using the rollout restart command) if the Secret data is consumed using env. If it is consumed via the filesystem, then no restart is required as Kubelet generally refreshes filesystems quickly.

Probes

Probes in Kubernetes are essential for maintaining the health and reliability of applications in a cluster. Probes serve as automated checks to determine if an application inside a pod is running correctly. If an application is not responsive, then when to restart a container, essentially self-healing the application. Probes also ensure that traffic is sent to a pod only when it’s ready to handle requests, preventing downtime and improving the user experience. Probes are useful if a slow-starting application is up, providing a buffer for applications that take longer to become ready, ensuring they aren’t prematurely restarted or marked as failed.

Kubernetes offers three types of probes: startup, readiness, and liveness.

startupProbe

In Kubernetes, startupProbe plays a pivotal role, particularly for applications that are slow to start, such as large legacy applications or complex Java-based applications. For instance, if you have a Java application that takes a considerable amount of time to boot up, without a startupProbe, Kubernetes might prematurely conclude that the application has failed and restart it, leading to a loop of unsuccessful attempts. The startupProbe acts as an initial gatekeeper, allowing the application a specified amount of time to become operational before the Kubernetes system begins performing regular health checks. This ensures that the application has adequate time to start up and stabilize without being interrupted by Kubernetes’ automatic recovery mechanisms. Please refer to the brief YAML snippet in Figure 9.13, illustrating the use of startupProbe in a Kubernetes Pod configuration:

Figure 9.13: Probe - Startup

The preceding snippet defines a startupProbe for a Kubernetes application. Here’s how it works:

Probe Type: The probe uses an HTTP GET request to check the application’s health. It targets the /actuator/health/startup endpoint, which is a common path used in Spring Boot applications for health checks.

Initial Delay: initialDelaySeconds: 30 means the probe starts 30 seconds after the container has begun. This delay gives the application enough time to initialize before the probe checks its status.

Check Frequency: periodSeconds: 10 indicates that the probe will check the application’s status every 10 seconds after the initial delay.

Success Criteria: successThreshold: 1 signifies that the probe needs only one successful check to consider the application started successfully.

Failure Tolerance: failureThreshold: 10 means Kubernetes will attempt the startup probe up to 10 times. If the probe fails 10 times, it will consider the application failed to start, and the container will be restarted.

In essence, this startupProbe is particularly useful for applications that require a significant amount of time to start, ensuring Kubernetes doesn’t restart them prematurely while they are still initializing.

To test this, run the following command:

kubectl apply -f \

https://raw.githubusercontent.com/brainupgrade-in/book-kubernetesadmin/main/chapter_09/05b-probes-springboot.yaml

Also, use the Kubernetes watch command to observe the behavior (Figure 9.14). Note that the application was restarted by Kubernetes (see the portion highlighted in red border). To simulate the starting of this application, the following endpoint can be used:

curl <pod-ip>:8080/switch/health/startup

You can get the pod IP using the following command:

kubectl get pods -owide -l app=probes-springboot

Soon after hitting the /switch/health/startup endpoint, the application is ready to serve requests.

Figure 9.14: Testing startupProbe

Please note that while the pod is not discoverable through the service; however, using pod IP, one can always access the application. This way of accessing the application should be limited, say for admin purposes only.

readinessProbe

In Kubernetes, the readinessProbe is a key tool used to determine if a container is ready to start accepting traffic. This probe checks the specified endpoint or command in the container and, if successful, informs the system that the container can be added to the service load balancer for handling requests. It is especially useful during deployments and scaling operations, as it ensures traffic is only sent to pods that are fully prepared to handle it. Unlike the startupProbe, which checks if a container main application is started, the readinessProbe specifically checks if it’s ready for operations, preventing premature traffic routing to new or restarted instances. In short, readinessProbe plays a crucial role in maintaining smooth user experience and operational stability in dynamic Kubernetes environments.

Please refer to the following YAML snippet that configures readinessProbe in a Kubernetes deployment:

Figure 9.15: Probe - readinessProbe

Probe Type: The readinessProbe is set to use an HTTP GET request. It targets the /actuator/health/readiness endpoint. This is typically used in Spring Boot applications to indicate application readiness.

Initial Delay: initialDelaySeconds: 5 specifies that the readiness check will start 5 seconds after the container has begun. This gives the application some time to initialize.

Probe Frequency: periodSeconds: 5 means the probe will check the readiness of the container every 5 seconds. This regular check is crucial for ensuring that the service only receives traffic when it’s truly ready.

Success Criterion: successThreshold: 1 implies that only one successful check is needed for the probe to consider the application ready. This helps in quickly routing traffic to the pod once it’s prepared.

Failure Tolerance: failureThreshold: 1 indicates that if the readiness probe fails even once, the container will be marked as not ready. This is stringent but ensures that traffic is only sent to fully operational containers.

In practice, this readinessProbe configuration is vital for maintaining service reliability and ensuring that no premature traffic is directed to a pod that isn’t fully ready to handle requests.

Run the following command to launch the application named probes-springboot:

kubectl apply -f \

https://raw.githubusercontent.com/brainupgrade-in/book-kubernetesadmin/main/chapter_09/05b2-probes-springboot.yaml

This application serves requests from the Kubernetes if it is connected to a rabbitmq; otherwise, it does not. Hence, run the following command to launch the redis database:

kubectl apply -f \

https://raw.githubusercontent.com/brainupgrade-in/book-kubernetesadmin/main/chapter_09/05a-rabbit.yaml

Please refer to Figure 9.16 to verify that the app is in ready status (1/1) once rabbitmq is launched.

Figure 9.16: Probe - readinessProbe successful

livenessProbe

In Kubernetes, the livenessProbe is a critical mechanism used to ensure the health and proper functioning of containers in a pod. It periodically checks whether the application inside the container is still running as expected. If the liveness check fails, Kubernetes interprets this as the application being in a broken state and responds by automatically restarting the faulty container. This process is key for self-healing in Kubernetes, as it helps to automatically recover from various failure scenarios such as deadlocks or memory leaks. Essentially, livenessProbe acts as an automated supervisor, ensuring that unhealthy containers are promptly refreshed to maintain smooth and continuous service operations.

Please refer to the YAML snippet in Figure 9.17 that configures livenessProbe in a Kubernetes deployment:

Figure 9.17: Probe - livenessProbe

Probe Mechanism: The livenessProbe is configured to perform an HTTP GET request to the /actuator/health/liveness endpoint. This endpoint is commonly used in Spring Boot applications for health checks.

Initial Delay: initialDelaySeconds: 60 means the probe starts checking the health of the application 60 seconds after the container has started. This delay provides sufficient time for the application to initialize.

Probe Interval: periodSeconds: 30 sets the frequency of the liveness probe to every 30 seconds. This interval is crucial to regularly check the health of the application.

Success Criteria: successThreshold: 1 specifies that only one successful probe is needed to consider the container alive. This ensures quick detection of a healthy container state.

Failure Tolerance: failureThreshold: 6 indicates that Kubernetes will attempt the liveness probe six times before concluding the container is not alive. Upon reaching this threshold, Kubernetes will restart the container, aiming to restore its normal operation.

This livenessProbe configuration plays a pivotal role in maintaining application reliability in Kubernetes, ensuring that any containers not running properly are automatically restarted to recover from potential failures.

To summarize, only after startupProbe succeeds, readinessProbe and livenessProbe are started and in parallel. Kubernetes does not restart the container but instead does not route requests to the container if its readinessProbe fails. In other probes, the container will be restarted if the probe fails.

Probe Mechanisms

HTTP GET Probe: Makes an HTTP GET request to the container’s IP address, a specified port, and path. Kubernetes considers the container healthy if the response code is in the range 200-399.

Figure 9.18: Probe type - httpGet

TCP Socket Probe: Tries to open a TCP connection to a specified port of the container. Success in opening a connection means the container is healthy.

Figure 9.19: Probe type - tcpSocket

Exec Probe: Executes a specified command inside the container. The container is considered healthy if the command exits with a status code of 0.

Figure 9.20: Probe type - exec

gRPC Probe: In Kubernetes, the gRPC probe mechanism adds another dimension to how we ensure the health and efficiency of containerized applications. This mechanism performs a remote procedure call using gRPC, a modern, high-performance RPC framework. For the gRPC probe on any particular port to work effectively, the target container must implement gRPC health checks as part of its service. The health of the container is evaluated based on the response status of these checks; if the response status is ‘SERVING’, Kubernetes considers the diagnostic as successful. This feature is particularly beneficial in microservices architectures where gRPC is commonly used, providing a more nuanced and service-specific health assessment compared to traditional HTTP or TCP checks.

Conclusion

In this chapter, we delved into configuring applications for production in Kubernetes, highlighting ConfigMaps for external configurations, Secrets for securing sensitive data, and Probes (liveness, readiness, and startup) for ensuring application health and traffic management. It emphasizes best practices in orchestration to enhance application reliability, security, and efficiency in production environments.

In the next chapter, we will navigate through the critical process of protecting your Kubernetes cluster. We will focus on the ETCD database, the cornerstone of your cluster’s data, exploring effective strategies for backup and restoration. Understanding ETCD’s integral role, we will also delve into the usage of ETCD CTL, a key tool for managing cluster data. This chapter is indispensable, especially when navigating complex tasks such as cluster upgrades, ensuring your cluster’s resilience and dependability.

Points to Remember

ConfigMaps are crucial for handling external configuration data, allowing applications to adapt to different environments without rebuilding images.

Secrets ensure sensitive data like passwords and tokens are stored and accessed securely, maintaining the integrity and confidentiality of the application.

Liveness Probes play a vital role in self-healing by restarting containers that are no longer functioning correctly.

Readiness Probes are essential for traffic management, ensuring that only pods ready to serve requests receive traffic, thus maintaining smooth user experiences.

Startup Probes provide a buffer for applications with slow start times, preventing them from being killed by liveness probes before they are fully ready.

The chapter underscores the importance of correctly configuring these elements to achieve a resilient, secure, and efficient production deployment in a Kubernetes environment.

It’s crucial to understand the right use cases and configurations for each of these components to maximize the benefits of Kubernetes’ orchestration capabilities.

Properly configured probes and management of configuration data directly contribute to the reliability and stability of applications in a Kubernetes cluster.

The chapter equips practitioners with the knowledge needed to handle application configurations and readiness checks, aligning with best practices in cloud-native development.

Ultimately, these tools and techniques are fundamental in optimizing Kubernetes deployments for the demanding requirements of production environments.

Multiple Choice Questions

What is the purpose of a ConfigMap in Kubernetes?

To manage sensitive data

To store external configuration data for applications

To create new pods

To monitor the health of applications

Where should sensitive data like passwords be stored in Kubernetes?

In a Deployment

In a Service

In a ConfigMap

In a Secret

What does a liveness probe determine in a Kubernetes pod?

If the pod can accept traffic

If the pod is running and healthy

The startup time of the pod

The configuration settings of the pod

What is the role of a readiness probe in Kubernetes?

To restart containers when they crash

To check if a container is ready to serve traffic

To secure container communication

To provide persistent storage

Which probe is used to give containers sufficient time to start in Kubernetes?

Liveness probe

Readiness probe

Startup probe

ConfigMap probe

How are ConfigMaps typically used in a Kubernetes environment?

To store and manage application binaries

To store non-sensitive configuration data for applications

To manage user roles and permissions

To automate application deployments

What happens if a liveness probe fails in Kubernetes?

The container is restarted

Traffic is no longer routed to the pod

The pod is permanently deleted

A new pod is created

When is a Kubernetes Secret required?

For health checking

For auto-scaling applications

For storing sensitive information like API keys

For network policy configurations

What is the key difference between a readiness probe and a liveness probe?

A readiness probe checks if the application is alive; a liveness probe checks if it’s ready to serve traffic

A liveness probe checks if the application is alive; a readiness probe checks if it’s ready to serve traffic

Both probes serve the same purpose

Readiness probes restart the container upon failure, while liveness probes do not

What should you use to manage environment variables for different deployment environments in Kubernetes?

Secrets

Services

ConfigMaps

VolumeMounts

Answers

b

d

b

b

c

b

a

c

b

c

Questions

How do you balance the convenience of using ConfigMaps for external configuration against the risk of overexposing non-sensitive yet crucial operational data?

In what scenarios might the default behavior of Secrets (not automatically updating when the Secret changes) impact the deployment or operation of a Kubernetes application?

Considering the crucial role of liveness and readiness probes in maintaining service availability, how would you design these probes for a complex application to avoid false positives and negatives in health checks?

How can startup probes be effectively utilized in applications with varying startup times across different environments, such as development, staging, and production?

What strategies would you employ to manage and rotate sensitive information in Secrets to adhere to security best practices without causing disruptions in service?

Keywords

ConfigMap

Secret

startupProbe

readinessProbe

livenessProbe

Probe Mechanisms - httpGet, tcp, exec, gRPC

CHAPTER 10

Cluster Database - Backup and Restore

Introduction

In the previous chapter, we focused on setting up applications for production in Kubernetes. It covers how ConfigMaps manages regular settings and how Secrets handles confidential information. The chapter also explains liveness, readiness, and startup probes, which check if applications are running correctly and ready for traffic. Emphasizing these tools’ importance for secure and efficient applications, it offers practical tips for their use, helping you to improve how they set up and manage their applications in Kubernetes for real-world scenarios.

In this chapter, we will guide you through essential practices for safeguarding your Kubernetes cluster, focusing on the ETCD database – the core repository of your cluster’s state and configurations. This chapter is focused on creating ETCD backup and restoration, crucial for cluster health, especially during upgrades. We will delve deep into the functionalities and strategic use of ETCD CTL, equipping you with the necessary skills to maintain, manage, and restore your cluster’s data, ensuring its resilience and robustness in various operational scenarios.

Examples referred to in this chapter are run in the multi-nodes cluster that is set up using the KIND method, giving cluster.yaml as an input configuration, as explained in Chapter 2, Installing Kubernetes.

Structure

In this chapter, we will cover the following topics:

Understanding ETCD

ETCD Cluster Configuration and Management

Backup and Disaster Recovery

Scaling ETCD Clusters

Security

Understanding ETCD

ETCD is a distributed key-value store, providing a reliable way to store data across a cluster of machines. It’s open-source and was originally designed by CoreOS. Its primary job is to safely store critical data for distributed systems and ensure that this data remains consistent across the cluster.

Core Features of ETCD

Following are some core features of ETCD:

Reliability: ETCD is built to be fault tolerant and resilient to failures, a non-negotiable attribute for Kubernetes, which relies on ETCD for cluster coordination and state management.

Consistency: It uses the Raft consensus algorithm to maintain consistency across a cluster. This ensures that each ETCD member reflects the same view of the cluster at any given time.

Availability: Through its distributed nature, ETCD is designed for high availability, mitigating single points of failure.

ETCD’s Role in Kubernetes Cluster Management

In Kubernetes, ETCD plays the role of the primary datastore for all cluster data. It stores the configuration data, state, and metadata, which Kubernetes needs to function. This includes data about the cluster’s components, such as nodes, pods, services, and more. The Kubernetes API server queries ETCD to retrieve the state of these objects and makes decisions based on this data.

ETCD’s role in Kubernetes is more than just a datastore; it’s the backbone that holds the cluster’s desired state and its real-time state. Kubernetes continuously monitors this state and works to reconcile any differences, ensuring the cluster’s health and functionality.

ETCD’s Consensus Algorithm (Raft)

Raft Basics: Raft is a consensus algorithm that is used by ETCD to manage a highly available replicated log. It ensures that each ETCD instance (or member) maintains an identical copy of this log.

Leader Election: Raft handles leader election. In an ETCD cluster, one member is elected as the leader, which handles all client requests and replicates data to the follower nodes.

Data Consistency: Raft guarantees that all changes to ETCD (like write operations) are agreed upon by a majority of nodes before they are committed. This ensures data consistency across the cluster.

Data Consistency and Reliability

ETCD ensures strong data consistency, which is crucial for the reliable operation of a Kubernetes cluster:

Consistency Guarantees: ETCD uses the Raft consensus algorithm to maintain a consistent state across all cluster nodes.

Fault Tolerance: ETCD’s distributed nature allows it to tolerate hardware failures and network partitions, ensuring high availability of data.

How Kubernetes Uses ETCD

Kubernetes relies on ETCD as its primary datastore for all cluster states and configurations. Here is how this interaction works:

Storage of Cluster State: ETCD holds the current state of all Kubernetes resources like pods, services, and deployments.

Resource Configuration Data: All configuration details, such as desired states and resource specifications, are stored in ETCD.

Cluster Coordination: ETCD plays a key role in cluster scheduling and orchestration. The scheduler and controller manager use ETCD data to make decisions.

Understanding Data Stored in ETCD

ETCD stores a variety of data essential for Kubernetes operations, such as:

Pods: Information about each pod’s state, specifications, and configuration.

Services: Details of service types, load balancing configurations, and service endpoints.

ConfigMaps and Secrets: Configuration parameters, sensitive information like passwords, and other configuration data can be dynamically loaded into pods.

Network Policies and Ingress Rules: Network configurations and rules governing how pods communicate with each other and the outside world.

Data Model and Key Schemas

ETCD’s data model is key-value based, with a hierarchical structure for keys. Let’s decode this:

Key Hierarchy: ETCD keys are often structured in a directory-like hierarchy, allowing efficient querying and organization of Kubernetes data.

Versioning and Watch Mechanism: ETCD keys are versioned, which is useful for concurrency control and watching for changes. Kubernetes components ‘watch’ keys for changes to trigger actions.

ETCD Cluster Configuration and Management

In this section, let us briefly look at the setup and configuration aspects of ETCD, especially in the context of Kubernetes.

Setting Up an ETCD Cluster

Installation Options: ETCD can be installed as a binary, as a Docker container, or using a package manager. Choose the method that best fits your cluster environment.

Initial Configuration: Key configurations include specifying member nodes, setting unique names for each member, and defining the initial cluster state.

Network Configuration: Properly configuring network settings is crucial for cluster communication. This includes client-to-server and server-to-server communication ports.

Configuring ETCD for Kubernetes

Kubernetes API Server Integration: The Kubernetes API server needs to be configured to communicate with the ETCD cluster. This involves specifying the ETCD endpoints in the Kubernetes API server startup configuration.

Data Directory: Set up and configure the data directory for ETCD, where all cluster data will be stored. Ensure it is secure and has sufficient space.

Time-To-Live (TTL) Settings: Understand and configure TTL for keys as needed, which determines how long data stays in the ETCD store.

In our case, we used KIND (Kubernetes setup using Docker containers), which in turn uses kubeadm to configure the Kubernetes cluster. The primary focus of the CKA exam is on the backup and restore of ETCD, including troubleshooting related to ETCD. Considering this, let us note down a few important details regarding its installation and configuration. To enter inside the control-plane node, you can use the command: docker exec -it cka-book-control-plane bash (where cka-book-control-plane is the name of the control plane node of your cluster), as shown in Figure 10.1 in the red color box.

Figure 10.1: ETCD - Command line arguments

Where:

ETCD is running as a static POD (blue-colored box).

ETCD process, including its arguments, is as follows (referring to the green-colored box):

root         647  2.1  0.3 11215488 56336 ?      Ssl  10:11   1:00 etcd --advertise-client-urls=https://172.18.0.3:2379 --cert-file=/etc/kubernetes/pki/etcd/server.crt --client-cert-auth=true --data-dir=/var/lib/etcd --experimental-initial-corrupt-check=true --experimental-watch-progress-notify-interval=5s --initial-advertise-peer-urls=https://172.18.0.3:2380 --initial-cluster=cka-book-control-plane=https://172.18.0.3:2380 --key-file=/etc/kubernetes/pki/etcd/server.key --listen-client-urls=https://127.0.0.1:2379,https://172.18.0.3:2379 --listen-metrics-urls=http://127.0.0.1:2381 --listen-peer-urls=https://172.18.0.3:2380 --name=cka-book-control-plane --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt --peer-client-cert-auth=true --peer-key-file=/etc/kubernetes/pki/etcd/peer.key --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt --snapshot-count=10000 --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt

The following are the important points to note:

The directories used for keeping the configuration and installation files are with respect to the kubeadm setup tool, and other tools may use a different approach.

Static pod definitions are kept at /etc/kubernetes/manifests, and this location is defined in the kubelet config file kept at /var/lib/kubelet/config.yaml.

Certificates for various control plane components are kept at /etc/kubernetes/pki.

Port used by ETCD to listen on 2379.

The data directory is specified using the argument: --data-dir=/var/lib/etcd.

Backup and Disaster Recovery

In this section, we will guide you through the process of creating backups and implementing disaster recovery strategies for ETCD, a critical skill for Kubernetes administrators and CKA exam aspirants.

Importance of ETCD Backup

ETCD holds all the state and configuration data of a Kubernetes cluster. Losing this data can result in catastrophic failures. Regular backups are vital for:

Disaster Recovery: In case of data corruption, accidental deletions, or hardware failures.

Cluster Upgrades: Safeguarding data before performing cluster upgrades or maintenance.

ETCD Data Manipulation and Access

Understanding how to interact with ETCD data is essential for the following reasons:

Using etcdctl Tool: etcdctl is the command-line tool for interacting with ETCD, allowing data retrieval and manipulation.

API Access: Kubernetes components interact with ETCD through a gRPC-based API, which offers efficient, language-agnostic data access.

Accessing ETCD with etcdctl

etcdctl is the command-line utility for interacting with ETCD. It’s essential for various operations including fetching data, adding new values, or monitoring the health of the cluster.

Setting Up etcdctl

As part of the CKA exam setup, etcdctl is generally available on the control node. If not, you can install it using this page etcd.io (https://etcd.io/docs/v3.5/install/). You will need to set environment variables like ETCDCTL_API to 3 for the API version and provide the necessary certificates if your ETCD is TLS-enabled.

Since, we are using the KIND setup, we can access the ETCDCTL to print the version as follows (Figure 10.2):

kubectl exec -n kube-system etcd-cka-book-control-plane -- sh -c “ETCDCTL_API=3 etcdctl --endpoints localhost:2379 --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key --cacert=/etc/kubernetes/pki/etcd/ca.crt version”

Figure 10.2: ETCDCTL - version

All etcdctl commands in this chapter are executed using this approach; however, to maintain brevity, command text starting with ETCDCTL_API=3 is mentioned.

Basic Commands

We can run an example pod using the following command:

kubectl run my-pod --image brainupgrade/hello

Now to retrieve the value of a key, my-pod in this case, use the following command:

ETCDCTL_API=3 etcdctl --endpoints localhost:2379 --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key --cacert=/etc/kubernetes/pki/etcd/ca.crt get /registry/pods/default/my-pod

Keys in ETCD for Kubernetes are often structured in a specific schema. For example, keys for pods are typically stored under /registry/pods/

Set a Key: To set a key-value pair, use the following commands:

ETCDCTL_API=3 etcdctl --endpoints localhost:2379 --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key --cacert=/etc/kubernetes/pki/etcd/ca.crt put /mykey “my value”

Delete a Key: To delete a key, use the following commands:

ETCDCTL_API=3 etcdctl --endpoints localhost:2379 --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key --cacert=/etc/kubernetes/pki/etcd/ca.crt delete /mykey

Be cautious when writing or deleting data directly to ETCD, as improper changes can disrupt the Kubernetes cluster.

Common etcdctl Commands and Operations

Listing Keys: To list all keys within a range:

ETCDCTL_API=3 etcdctl --endpoints localhost:2379 --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key --cacert=/etc/kubernetes/pki/etcd/ca.crt get --prefix /registry/pods

Health Check: To check the health of the ETCD cluster:

ETCDCTL_API=3 etcdctl --endpoints localhost:2379 --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key --cacert=/etc/kubernetes/pki/etcd/ca.crt endpoint health

Step-by-Step Guide to Backing Up and Restoring ETCD

Before taking a backup, let us create a backup directory to store the backup data. Use the following command:

mkdir /var/lib/etcd/backup

Backing Up ETCD

Creating a backup of ETCD data involves taking a snapshot of the ETCD database.

Use the etcdctl snapshot save command to create a snapshot, as follows:

ETCDCTL_API=3 etcdctl --endpoints localhost:2379 --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key --cacert=/etc/kubernetes/pki/etcd/ca.crt snapshot save /var/lib/etcd/backup/snapshot.db

This command creates a snapshot of the ETCD database and saves it to the specified path.

To verify the snapshot, run the following command:

ETCDCTL_API=3 etcdctl --endpoints localhost:2379 --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key --cacert=/etc/kubernetes/pki/etcd/ca.crt --write-out=table snapshot status /var/lib/etcd-backup

Refer to Figure 10.3 as a reference:

Figure 10.3: ETCD backup

ETCD Data Restoration

We had launched my-pod earlier, so let us delete this pod and then initiate the recovery by using the following command:

kubectl delete pod my-pod

Now, let us restore the snapshot - earlier state of ETCD, that is, before the pod was deleted. Restoring ETCD from a snapshot is crucial in a disaster recovery scenario. Here are the steps:

First, create the restoration directory:

mkdir /var/lib/etcd/restore

Then, run the following command:

ETCDCTL_API=3 etcdctl --endpoints localhost:2379 --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key --cacert=/etc/kubernetes/pki/etcd/ca.crt --data-dir=/var/lib/etcd/restore snapshot restore /var/lib/etcd/backup/snapshot.db

Please note that we gave --data-dir=/var/lib/etcd/restore along with the location where the snapshot is kept. Refer to Figure 10.4 for reference:

Figure 10.4: ETCD restore

ETCD snapshot is restored at /var/lib/etcd/restore, and we need to provide this input to the ETCD process. As mentioned earlier, ETCD is launched as a static pod and its definition is kept at /etc/kubernetes/manifest/etcd.yaml. Modify this YAML file and change the data directory to /var/lib/etcd/restore.

The control plane node may not have a file editing utility, and if that is the case, you can install one (see Figure 10.5):

apt-get update && apt-get install -y vim

vim /etc/kubernetes/manifest/etcd.yaml

Figure 10.5: ETCD manifest - data directory update

Once etcd.yaml is modified, kubelet will pick up the changes in the manifest file and update the ETCD process. It may take some time, so don’t panic, just be patient and try listing the pods again. You will notice that my-pod is running, and this proves that ETCD backed-up data is fully restored and verified too.

Best Practices for ETCD Backups:

Regular and Frequent Backups: Schedule backups based on the frequency of changes in your cluster.

Offsite Storage: Store backups in a secure, offsite location to protect against data center failures.

Backup Validation: Regularly test backups to ensure they are valid and can be restored successfully.

It’s recommended to automate the backup process. You can use cron jobs or Kubernetes cronjobs for scheduling regular backups, including its validation.

Scaling ETCD Clusters

While ETCD is not typically a bottleneck in Kubernetes, scaling becomes necessary as the workload and size of the cluster grow.

Vertical Scaling: This involves increasing the resources (CPU, memory, disk) of existing ETCD nodes. Most often, scaling ETCD is more about improving the disk performance rather than just adding more CPUs or memory.

Horizontal Scaling: Adding more members to the ETCD cluster. However, note that beyond a certain point, adding more members can actually degrade performance due to the increased communication overhead.

Security

Given that ETCD holds critical data, securing it is non-negotiable. In the context of a Kubernetes cluster, this means securing both the communication with the ETCD server and the data stored within it. Communication with ETCD should be encrypted using TLS, preventing unauthorized access and ensuring data integrity.

For data at rest, ETCD supports encryption, which means that all data stored in the ETCD database can be encrypted using a key. This adds an extra layer of security, ensuring that even if the data is somehow accessed, it remains unreadable without the proper decryption key.

Access control is another crucial aspect of securing ETCD. Kubernetes administrators should implement role-based access control (RBAC) to ensure that only authorized users and processes can access or modify the ETCD data. This prevents accidental or malicious changes that could disrupt the cluster’s functionality.

Conclusion

In this chapter, we delved deep into the functionalities and management of ETCD in Kubernetes, highlighting its pivotal role in maintaining cluster state and configuration. We covered essential topics such as setting up, securing, and scaling ETCD clusters, and the intricacies of data management within Kubernetes. The chapter also provided practical insights into using etcdctl for ETCD interaction, emphasizing the importance of regular backups. Additionally, we explored high availability, performance optimization, and common challenges associated with ETCD. Tailored for Kubernetes administrators and CKA exam candidates, this chapter offered a blend of theoretical knowledge and practical application essential for mastering Kubernetes infrastructure.

In the next chapter, we will delve into the advanced techniques for upgrading Kubernetes clusters. The focus will be on utilizing kubeadm to effectively update both the control plane and worker nodes. Key elements include understanding version skew policy and ensuring compatibility during upgrades. The chapter will also emphasize the importance of post-upgrade checks and elaborate on robust rollback strategies to maintain the cluster’s integrity and stability after upgrades.

Points to Remember

ETCD’s Central Role: Understand the pivotal role of ETCD as a distributed key-value store in Kubernetes, crucial for maintaining cluster state and configurations.

Data Consistency and Reliability: Recognize the importance of ETCD’s features, such as fault tolerance, data consistency, and high availability, facilitated by the Raft consensus algorithm.

Data Storage and Retrieval: Be familiar with the types of data stored in ETCD, such as information about pods, services, ConfigMaps, and network policies, and how Kubernetes uses this data.

Cluster Configuration Management: Gain insights into the proper setup and configuration of ETCD in a Kubernetes environment, including installation options and network configurations.

Backup and Disaster Recovery: Emphasize the criticality of regular backups for ETCD, covering aspects of disaster recovery, data manipulation, and cluster upgrades.

Using etcdctl Tool: Develop proficiency in using etcdctl for ETCD data interaction, including fetching, adding, and monitoring data.

Backup and Restoration Process: Master the steps for backing up and restoring ETCD, including snapshot creation, verification, and restoration techniques.

Securing ETCD: Prioritize securing ETCD with measures like TLS for communication, encryption for data at rest, and implementing robust access control mechanisms.

Multiple Choice Questions

What is the primary role of ETCD in Kubernetes?

Load balancing

Data storage for cluster state and configuration

Network traffic routing

Container orchestration

Which consensus algorithm does ETCD use to maintain data consistency?

Paxos

Raft

Zab

Gossip

What is the recommended number of ETCD members for a high availability setup?

Any even number

5

3

1

Which command is used to check the health of an ETCD cluster?

etcdctl check health

etcdctl status

etcdctl endpoint health

etcdctl cluster health

In ETCD, what is the purpose of a snapshot?

To increase the performance of the ETCD cluster

To backup ETCD data

To restore lost data

To create a new ETCD cluster

For ensuring ETCD data integrity, what is an essential practice?

Regularly updating Kubernetes

Frequent rebooting of ETCD nodes

Regularly backing up ETCD data

Increasing the number of ETCD nodes

What is a common method to optimize ETCD performance?

Decreasing the snapshot interval

Compacting the ETCD database

Disabling TLS encryption

Increasing the number of client requests

Which of the following is NOT stored in ETCD in a Kubernetes cluster?

Pod specifications

Service configurations

Container images

Network policies

Answers

b

b

c

c

b

c

b

c

Questions

ETCD Data Consistency: How does the Raft consensus algorithm ensure data consistency in ETCD, and why is this important for the overall health and functionality of a Kubernetes cluster?

Backup Strategies: What are the best practices for implementing ETCD backup strategies in a Kubernetes environment, and how do these practices ensure data integrity and disaster recovery?

Restoration Techniques: In what scenarios would you need to restore ETCD from a backup, and what are the key steps and considerations involved in the restoration process?

Scaling ETCD Clusters: What factors should be considered when scaling an ETCD cluster in Kubernetes? How do vertical and horizontal scaling impact the cluster’s performance and reliability?

Security Measures: What specific security measures should be implemented to protect ETCD in a Kubernetes cluster, and how do these measures contribute to the overall security posture of the cluster?

Key Terms

ETCD: A distributed key-value store used by Kubernetes to store all cluster data, including state and configurations.

Raft Consensus Algorithm: A consensus algorithm used by ETCD to ensure data consistency across the cluster by managing a replicated log.

etcdctl: The command-line tool for interacting with ETCD, crucial for data retrieval, manipulation, and administrative tasks.

Snapshot: A point-in-time capture of the ETCD datastore, used for creating backups and ensuring data integrity.

Cluster State: Refers to the current status and configuration of all resources within a Kubernetes cluster, as stored in ETCD.

Data Restoration: The process of returning ETCD to a previous state using a backup, typically employed during disaster recovery.

Key-Value Store: The fundamental data model of ETCD, where data is stored as key-value pairs.

Backup Validation: The process of verifying that an ETCD backup is complete, accurate, and can be successfully restored.

Horizontal Scaling: Adding more nodes to the ETCD cluster to distribute the load and increase redundancy.

TLS Encryption: Refers to the use of Transport Layer Security to secure communications with the ETCD server, protecting data in transit.

CHAPTER 11

Cluster Upgrade – kubeadm

Introduction

The last chapter provided a comprehensive overview of ETCD in Kubernetes, focusing on its role as a critical component for storing and managing cluster state and configuration. Key aspects included ETCD’s setup, configuration, and the importance of securing and regularly backing up the ETCD data. We also explored strategies for ensuring high availability and scalability of ETCD clusters. Practical tips on using the etcdctl tool for effective ETCD management were highlighted.

Kubernetes, an open-source container orchestration platform, is continually evolving with new features, bug fixes, and security enhancements. Upgrading your Kubernetes cluster ensures you benefit from these improvements, making your system more efficient, secure, and feature-rich. This chapter guides through a detailed, step-by-step process of upgrading both the control plane and worker nodes, using kubeadm. It also addresses the critical aspects of version skew policy and compatibility. Finally, it covers post-upgrade verification and rollback strategies, ensuring the cluster’s integrity and stability post-upgrade.

The examples referred to in this chapter are run in the multi-node cluster that is set up using the KIND method, giving cluster.yaml as an input configuration, as explained in Chapter 2, Installing Kubernetes.

Structure

In this chapter, we will cover the following topics:

Understanding Release Cycle

Preparing for Cluster Upgrade

Executing the Cluster Upgrade

Understanding the Release Cycle

Kubernetes follows a predictable release cycle, typically releasing three minor versions each year. Each release includes new features, scheduled deprecations, and bug fixes. Understanding this cycle helps in planning and preparing for upgrades.

Kubernetes versions are denoted as x.y.z, where ‘x’ is the major version, ‘y’ is the minor version, and ‘z’ is the patch version. Patch releases focus on bug fixes and security patches, while minor releases introduce new features and deprecations.

The Kubernetes project adheres to Semantic Versioning (SemVer). This scheme helps users understand the impact of upgrading to a new version.

Preparing for Cluster Upgrade

Before upgrading, assess your current cluster’s version and configuration. Use the kubectl version and kubectl get nodes to gather this information. Each Kubernetes release comes with detailed release notes. Review these to understand new features, deprecations, and any notable changes. Pay special attention to breaking changes that might affect existing deployments. In addition, explore new features that could benefit your system.

Backup Strategies: A comprehensive backup strategy is critical before any upgrade. This includes backing up ETCD data, application data, and cluster configurations.

Backing Up ETCD Data: ETCD, the primary datastore for Kubernetes, should be backed up. Use the ETCD’s built-in snapshot feature for this. Please refer to Chapter 10, Cluster Database - Backup and Restore, for greater details.

Ensuring Application Data Safety: Ensure your applications’ persistent data is backed up. This might involve backing up data in persistent volumes or using cloud-based storage snapshots.

Executing the Cluster Upgrade

This section provides a step-by-step guide to upgrading a Kubernetes cluster set up with KIND from version 1.28.0 to 1.28.4 using kubeadm. First, set up the KIND cluster. You can use the vagrant box (refer to Chapter 2, Installing Kubernetes) to run the following commands if you don’t have a Linux machine.

Installing KIND Utility

[ $(uname -m) = x86_64 ] && curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.20.0/kind-linux-amd64

chmod +x ./kind

sudo mv ./kind /usr/local/bin/kind

Launching the Cluster

kind create cluster --name cka-book --config cluster.yaml --image kindest/node:v1.28.0

The content of the cluster.yaml can be downloaded from the following location:

https://raw.githubusercontent.com/brainupgrade-in/book-kubernetesadmin/main/chapter_02/cluster.yaml

Once the command is executed, you can enter inside the control plane node using the following command (remember that Docker is essential for KIND setup):

docker exec -it cka-book-control-plane bash

Figure 11.1: KIND cluster

KIND makes use of kubeadm to set up the Kubernetes cluster. CKA exam expects the usage of kubeadm to upgrade the cluster. In order to upgrade the cluster, first run the following command to figure out which version the cluster can be upgraded to (see Figure 11.2):

kubeadm upgrade plan

Figure 11.2: kubeadm - upgrade plan

As you can see from Figure 11.2, in order to update the Kubernetes cluster to version 1.28.4, we need to first update the kubeadm utility. To update the kubeadm, first, we need to update the OS package index by including kubeadm as part of the index (if not available already). Using the following commands, the OS repository index can be updated:

apt-get update && apt install -y gpg

mkdir -p /etc/apt/keyrings

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.28/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

echo ‘deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.28/deb/ /’ | tee /etc/apt/sources.list.d/kubernetes.list

apt-get update

To know the exact version of kubeadm available in the OS repository, run the following command (see Figure 11.3):

apt search kubeadm

Figure 11.3: kubeadm - OS package version

Run the following command to update both kubeadm and kubelet:

apt-get install kubeadm=1.28.4-1.1 kubelet=1.28.4-1.1 -y

Once done, restart the kubelet, and you will notice that the node version is updated to v1.28.4 (see Figure 11.4):

service kubelet restart

kubectl get nodes

Figure 11.4: Node - kubelet version

Remember that we have not yet updated the main cluster components. The following command updates those components, thus completing the upgrade of the control plane node:

kubeadm upgrade apply v1.28.4

The preceding command pulls the required images and updates the static pod manifests as well, so that kubelet can restart the control plane components.

Figure 11.5: Cluster upgrade - SUCCESS

You may notice that the cluster becomes inaccessible for a brief period, which is expected.

After some time, try the following command, and you will notice that the control plane components are now running with new images (see Figure 11.6):

Figure 11.6: Control plane components - newer images

Upgrading Worker Nodes

Before we start the upgrade process, let us run a sample app using the following command. This will help us observe how Kubernetes moves user applications to other nodes when the current node is being upgraded:

kubectl create deploy hello --replicas 5 --image brainupgrade/hello

Before we start upgrading the node, cka-book-worker, let us take a look at the application pods and nodes they are currently running:

Figure 11.7: Hello App pods - nodes

First, we need to drain the node as part of the upgrade process. Refer to the following command:

kubectl drain cka-book-worker --ignore-daemonsets

This will evict all user applications (pods) to other available nodes. This is important from the availability perspective since we are expected to upgrade the cluster without disrupting the running applications (see Figure 11.8):

Figure 11.8: Draining of worker node

Now, log in to the worker node: cka-book-worker

docker exec -it cka-book-worker bash

Note: During the exam, you may use ssh to log in to the appropriate node.

Update the kubeadm and kubelet packages using the following commands:

apt-get update && apt install -y gpg

mkdir -p /etc/apt/keyrings

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.28/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

echo ‘deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.28/deb/ /’ | tee /etc/apt/sources.list.d/kubernetes.list

apt-get update

apt-get install kubeadm=1.28.4-1.1 kubelet=1.28.4-1.1 -y

Now, restart the kubelet and upgrade the node. Refer to the following commands:

systemctl restart kubelet

kubeadm upgrade node

Figure 11.9: Node upgrade

Now, exit from the worker node and uncordon this worker node from the control-plane node. Refer to the following commands:

docker exec -it cka-book-control-plane bash

kubectl uncordon cka-book-worker

If you don’t uncordon the worker node, new workloads will not be scheduled on this node (see Figure 11.10):

Figure 11.10: Operational worker node

Similarly, the second worker node, cka-book-worker2, can be upgraded without disrupting the current user workload (see Figure 11.11):

Figure 11.11: Cluster upgrade - complete

Version Skew and Compatibility Considerations

Understanding Kubernetes’ version skew policy is important for ensuring compatibility between the control plane and node components. The control plane must be at a higher or equal version than node components. The version skew should not exceed the +/-1 minor version.

Rollback and Troubleshooting

Consider the following in case any rollback or troubleshooting is required:

Rollback Strategies for Failed Upgrades: Have a clear rollback strategy in case of upgrade failure. This could include restoring etcd from a backup or downgrading Kubernetes components.

Restoring from Backups: In case of a critical failure, restore etcd data and application data from backups.

Downgrading Components: If rollback is required, follow Kubernetes’ guidelines for downgrading components to their previous versions.

Common Upgrade Issues and Troubleshooting Tips: Address common upgrade issues such as version incompatibilities, configuration errors, and resource limitations.

Handling Version Incompatibilities: Ensure all components and dependencies are compatible with the new Kubernetes version.

Dealing with Configuration and Resource Limitations: Adjust configurations and resource allocations as necessary to accommodate the upgraded environment.

Conclusion

In this chapter, we delved into the critical aspects of upgrading a Kubernetes cluster. We began with an overview of Kubernetes’ release cycle and versioning, followed by detailed preparations, including backup strategies and reviewing release notes. The chapter then provided a step-by-step guide to the upgrade process, covering both control planes and worker nodes utilizing kubeadm. Post-upgrade steps included thorough verification of cluster health and application functionality. Finally, we addressed rollback strategies and troubleshooting, concluding with best practices for smooth and effective cluster upgrades.

The upcoming chapter focuses on CoreDNS, a vital element for DNS resolution and service discovery in Kubernetes ecosystems. We will explore its architecture and operational strategies, highlighting CoreDNS’s crucial role in facilitating smooth communication within clusters. This segment will offer in-depth insights into CoreDNS’s functionalities and best practices, emphasizing its significance in Kubernetes networking and overall cluster management.

Points to Remember

Review Kubernetes Release Notes: Always review the release notes for the target version to identify new features, breaking changes, and deprecations.

Backup Before Upgrading: Ensure a comprehensive backup of ETCD data and application data before initiating an upgrade.

Sequentially Upgrade Control Plane and Nodes: Start by upgrading the control plane components with kubeadm, followed by a systematic upgrade of worker nodes.

Drain Nodes Before Upgrading: Safely evict all running pods by draining the nodes before upgrading them.

Verify and Monitor Post-Upgrade: After the upgrade, thoroughly verify the cluster’s and applications’ functionality and continuously monitor for any issues.

Multiple Choice Questions

What is the primary reason to upgrade a Kubernetes cluster?

To change its configuration

To improve performance and security

To rename the cluster

To reduce the number of nodes

Which command is used to drain nodes before upgrading them?

kubectl delete nodes

kubectl drain

kubectl upgrade nodes

kubectl evacuate

What should be reviewed before upgrading a Kubernetes cluster?

User manuals

Release notes of the target version

Old configuration files

What type of releases typically include new features in Kubernetes?

Major releases

Minor releases

Patch releases

Which of the following is crucial to perform before a Kubernetes upgrade?

Renaming all nodes

Backing up ETCD data

Reinstalling all applications

After upgrading the control plane, what is the next step?

Upgrading network policies

Upgrading worker nodes

Changing cluster names

What is the Kubernetes version skew policy concerning control plane and node versions?

The control plane must always be two minor versions ahead of node versions

Node versions must be higher than the control plane versions

The control plane must be at a higher or equal version than node components, and version skew should not exceed +/-1 minor version

The control plane must be at a higher or equal version than node components

In case of a failed upgrade, what is one of the recommended rollback strategies?

Increase the number of nodes

Restore from backups

Rename the cluster

Answers

b

b

b

b

b

b

d

b

Questions

What contingency plans and rollback procedures should be in place to handle potential failures or complications during the upgrade process?

How can you ensure minimal downtime and maintain workload stability during the node draining and upgrade process in a production environment?

What are the key learnings from the upgrade process that can be applied to future cluster management and maintenance strategies?

Keywords

Kubernetes Cluster Upgrade

Release Cycle

Minor, Major, and Patch Releases

Versioning Scheme

Backup Strategies

Control Plane Components

Worker Nodes

Draining Nodes

Version Skew Policy

Rollback Strategies

CHAPTER 12

CoreDNS

Introduction

The last chapter provided a comprehensive overview of ETCD in Kubernetes, focusing on its role as a critical component for storing and managing cluster state and configuration. The major aspects included ETCD’s setup, configuration, and the importance of securing and regularly backing up the ETCD data. We also explored strategies for ensuring high availability and scalability of ETCD clusters. Practical tips on using the etcdctl tool for effective ETCD management were highlighted.

In this chapter, we will discuss an essential component for DNS resolution and service discovery, CoreDNS which stands at the forefront of enabling seamless communication in Kubernetes clusters. This chapter delves into its architecture, operational strategies, and the pivotal role it plays in cluster management. These insights will provide a deeper understanding of CoreDNS’s functionalities, best practices, and its integral position in Kubernetes networking.

The examples referred to in this chapter are run in the multi-node cluster that is set up using the Kind method, giving cluster.yaml as an input configuration, as previously explained in Chapter 2: Installing Kubernetes.

Structure

In this chapter, we will cover the following topics:

Overview of CoreDNS

CoreDNS - Architecture and Components

CoreDNS - Deployment and Configuration

Customizing CoreDNS

DNS in Kubernetes Networking

Performance and Scalability

Security: Safeguarding Kubernetes DNS Services

Operational challenges - CoreDNS

Overview of CoreDNS

In the ever-evolving landscape of cloud computing and containerization, service discovery remains a critical component, particularly in Kubernetes environments. CoreDNS, a flexible and extensible DNS server, has emerged as a pivotal player in this space. Its role in Kubernetes is not just foundational but transformative, offering a fresh perspective on how services communicate in a clustered environment.

CoreDNS, at its core, is a DNS server that doubles as a service discovery mechanism. It’s designed to be lightweight and highly configurable, making it an ideal choice for modern, dynamic environments like Kubernetes. Unlike traditional DNS servers, CoreDNS is built with a plug-in architecture, allowing for extended functionality and customization. This architecture enables CoreDNS to support a wide range of DNS queries and respond dynamically based on the Kubernetes environment’s state.

In Kubernetes, CoreDNS plays a crucial role in service discovery, a mechanism that allows pods to locate and communicate with each other and external services. When a service is created in Kubernetes, CoreDNS updates its records, ensuring that any requests to that service are correctly routed. This seamless integration with Kubernetes makes CoreDNS an indispensable tool for ensuring efficient networking within clusters.

CoreDNS simplifies and streamlines cluster internal communication. Translating service names to IP addresses ensures that pod-to-pod communication is efficient and reliable. This functionality is crucial in a microservices architecture, where services are dynamically scaled and updated.

Comparison with Kube-DNS

Kube-DNS, the predecessor of CoreDNS in Kubernetes, laid the groundwork for service discovery within clusters. However, CoreDNS brings several advancements over Kube-DNS, including:

Extensibility: CoreDNS’s plug-in architecture allows for more customization and functionality, making it more adaptable to complex and varied use cases.

Performance: CoreDNS is known for its performance improvements, especially in large-scale clusters, where DNS queries are more frequent and demanding.

Simplicity: The configuration of CoreDNS tends to be simpler and more straightforward, making it easier for administrators to manage.

Flexibility: CoreDNS’s ability to support multiple DNS protocols and integrate with various backends makes it a more versatile choice for diverse Kubernetes environments.

CoreDNS Architecture and Components

CoreDNS is designed with a focus on flexibility, extensibility, and performance. Its architecture is unique, primarily due to its modular and plug-in-based design. Unlike traditional monolithic DNS servers, CoreDNS operates on a core binary, around which various plugins can be wrapped based on the specific needs of the environment. This design philosophy allows CoreDNS to be both lightweight and adaptable.

Key Components of CoreDNS

Following are some of the key componenets of CoreDNS:

Core Binary: At the heart of CoreDNS is its core binary, which is responsible for basic DNS serving capabilities. This binary is lean and optimized for performance.

Plugins: The true power of CoreDNS lies in its plugins. Each plugin adds a specific functionality to the DNS server. These can range from basic DNS features like caching and logging to more advanced capabilities like health checking and metrics gathering.

Configuration File (Corefile): CoreDNS is configured using a Corefile, which is a simple, yet powerful way to specify how CoreDNS should behave. Each server block in the Corefile corresponds to a DNS zone and contains directives and plugins that define how requests for that zone should be handled.

Service Discovery Mechanisms: CoreDNS can integrate with various service discovery backends, including Kubernetes, etcd, and others, allowing it to dynamically update DNS records based on the state of these systems.

Load Balancing and Health Checking: CoreDNS can perform load balancing of requests among multiple backend services and conduct health checks to ensure it directs traffic only to healthy nodes.

CoreDNS Integration with Kubernetes

CoreDNS is an integral part of Kubernetes, particularly for service discovery and internal DNS services. It replaces Kube-DNS in most modern Kubernetes setups. CoreDNS’s architecture and components make it an ideal choice for Kubernetes clusters, offering a flexible, scalable, and efficient solution for service discovery and DNS resolution. Its plug-in-based architecture, combined with its seamless integration with Kubernetes, positions CoreDNS as a key enabler in the Kubernetes ecosystem, facilitating smoother and more reliable service communication in cloud-native environments. Here’s how CoreDNS integrates with Kubernetes:

Service Discovery: CoreDNS watches the Kubernetes API for changes in services and endpoints. When a new service is created, CoreDNS automatically updates its DNS records, allowing pods to resolve the service name to its IP address.

Pod DNS Resolution: Pods in Kubernetes are configured to use CoreDNS for DNS resolution by default. This configuration allows pods to communicate with each other using service names instead of IP addresses, simplifying inter-service communication.

Scalability and Reliability: CoreDNS’s lightweight architecture and support for horizontal scaling make it well-suited for the dynamic nature of Kubernetes environments, where pod and service counts can change frequently.

Custom Resource Definitions (CRDs): CoreDNS can leverage Kubernetes CRDs to define custom DNS entries, providing more control over DNS resolution within the cluster.

Network Policies: CoreDNS works with Kubernetes network policies, ensuring that DNS queries are handled in accordance with the security and traffic rules defined for the cluster.

CoreDNS Deployment and Configuration

In most Kubernetes installations, CoreDNS is automatically deployed as a part of the cluster setup. For instance, tools like kubeadm handle the deployment of CoreDNS. CoreDNS has been the default DNS service in Kubernetes since version 1.11, replacing Kube-DNS. This shift was driven by CoreDNS’s enhanced capabilities, such as better performance, flexibility, and easier configurability. We can use the following command to know the details of its deployment (Figure 12.1):

kubectl describe deploy coredns -n kube-system

Figure 12.1: CoreDNS deployment

For custom deployments or specific needs, CoreDNS can be manually deployed. This involves setting up the CoreDNS Deployment and Service resources in Kubernetes and ensuring proper configuration. Post-deployment, validate that CoreDNS pods are running. This can be done using the following command:

kubectl get pods -n kube-system -l k8s-app=kube-dns -owide

Configuration

CoreDNS Configuration File (Corefile) is the primary configuration file for CoreDNS. It specifies how CoreDNS serves DNS requests. The Corefile is represented as a ConfigMap in Kubernetes, allowing dynamic updates without restarting pods. Run the following command to view the Corefile content (Figure 12.2):

kubectl get cm coredns -n kube-system -oyaml

Figure 12.2: CoreDNS – Corefile

Understanding Corefile File

The Corefile is the heart of CoreDNS’s configuration. It’s a powerful and flexible file that dictates how CoreDNS responds to DNS queries. The file is structured in a zone-based format, where each zone’s configuration includes a series of directives and plugins.

The structure of Corefile is as follows:

Zone Blocks: Define the DNS zones CoreDNS will serve. Each block can have multiple directives and plugins.

Directives: Commands that perform an action or modify the behavior of the server.

Plugins: Extend CoreDNS functionalities, handling tasks from simple DNS responses to complex manipulations based on Kubernetes services and pods.

Corefile typically contains the following plugins:

Forward Plugin: Routes DNS queries to specified upstream servers. Essential for integrating CoreDNS with external DNS servers for non-cluster domains.

Cache Plugin: Caches DNS query results to enhance performance. This plugin is critical in high-load environments to reduce response times and server load.

Log Plugin: Enables logging of DNS queries, providing insights into DNS traffic and potential issues.

Health Plugin: Offers a health check endpoint, allowing monitoring tools to verify the health of CoreDNS.

Kubernetes Plugin: Core to CoreDNS’s functionality in Kubernetes, this plugin enables DNS-based service discovery within the cluster.

Metrics Plugin: Exposes Prometheus metrics, essential for monitoring and analyzing the performance and health of CoreDNS.

Customizing CoreDNS

CoreDNS offers extensive customization options. Here are some examples:

Adding Custom DNS Entries: By modifying the Corefile, custom DNS records (such as A, AAAA, and CNAME records) can be added. This is particularly useful for defining shortcuts to external services or for defining records for services not managed by Kubernetes.

Advanced Forwarding: CoreDNS allows for sophisticated forwarding configurations, including conditional forwarding based on domain names or implementing load balancing across multiple upstream servers.

Integrating with External DNS Services: CoreDNS can be configured to work alongside external DNS services, facilitating seamless integration in hybrid cloud environments.

Configuring Health Checks and Load Balancing: CoreDNS can be customized to perform health checks on backend services and to balance the load among multiple endpoints, enhancing the reliability and efficiency of the DNS service.

When customizing CoreDNS, it’s crucial to consider the security implications, particularly in a Kubernetes environment. This includes ensuring secure communication between CoreDNS and Kubernetes API and protecting against DNS-based attacks.

You can update the ConfigMap using the following command:

kubectl edit configmap coredns -n kube-system

After editing the ConfigMap, CoreDNS pods need to be restarted to apply the changes. This can be done by deleting the CoreDNS pods and allowing Kubernetes to recreate them with the new configuration. Be prepared to troubleshoot issues like DNS resolution failures or performance bottlenecks.

Mastering CoreDNS configuration through the Corefile is essential for optimizing DNS performance in Kubernetes environments. The ability to customize CoreDNS with various plugins and directives enables it to cater to the specific needs of different cluster setups.

DNS in Kubernetes Networking

In Kubernetes, DNS is not just a utility but a fundamental part of the networking architecture. It enables pods to locate services and communicate with them seamlessly, which is essential in a microservices-based architecture.

Service Discovery: Kubernetes employs DNS for service discovery. When a service is created in Kubernetes, a DNS record is automatically generated. This allows pods to resolve the service name to its cluster IP.

CoreDNS is the default DNS service in Kubernetes that handles DNS queries within the cluster. It resolves the names of services and pods to their corresponding IP addresses. When a pod makes a DNS query (for example, accessing a service), the query is directed to CoreDNS. CoreDNS then checks its records and returns the IP address of the requested service.

DNS for Pod-to-Pod Communication: Beyond service discovery, DNS is also used for pod-to-pod communication. Pods can communicate with each other using DNS names, abstracting away the underlying IP addresses.

CoreDNS in Pods

Pods are configured to use CoreDNS by default. This is typically done through the kubelet, which sets the DNS server in the pod’s resolv.conf file to the IP address of the CoreDNS service. The following commands can be used to execute this:

First launch a web server using nginx image:

kubectl run webserver --image nginx

Check the nameserver IP in the resolv.conf placed inside the webserver pod:

kubectl exec webserver -- cat /etc/resolv.conf

Check the cluster IP of the CoreDNS service named kube-dns (Figure 12.3):

kubectl describe svc kube-dns -n kube-system | grep IP:

Figure 12.3: CoreDNS service IP in pod

Interaction with Other Components

CoreDNS and API Server: CoreDNS interacts closely with the Kubernetes API server. It watches for changes in services and endpoints and updates its DNS records accordingly.

Integration with Network Plugins: CoreDNS works in conjunction with various Kubernetes network plugins, such as Calico, Flannel, or Weave. These plugins manage the networking layer of Kubernetes, and CoreDNS complements them by resolving service names to the network addresses managed by these plugins.

Load Balancing and CoreDNS: CoreDNS plays a part in the load balancing of services in Kubernetes. When there are multiple pods for a service, CoreDNS can return different IP addresses for the same service name, distributing the load across these pods.

CoreDNS and Ingress Controllers: CoreDNS can also integrate with Kubernetes Ingress controllers. Ingress controllers manage external access to services in a cluster, and CoreDNS can resolve domain names to the Ingress resources, facilitating external communication.

In the interaction of CoreDNS with other components, security is paramount. Ensuring secure communication channels and proper authentication between CoreDNS and the Kubernetes API server is crucial to prevent DNS spoofing or hijacking.

Performance and Scalability

Performance and scalability are crucial for any technology in a Kubernetes environment. CoreDNS, as the default DNS service in Kubernetes, is no exception. Following are a few performance considerations, best practices for scaling, and monitoring and logging strategies for CoreDNS in a Kubernetes cluster.

Performance Considerations

Optimized Corefile Configuration: The configuration of CoreDNS via the Corefile is key to its performance. Optimize plugins’ usage, and tailor the configuration to the specific needs of your Kubernetes environment.

Caching Strategies: Utilize the cache plugin effectively. Proper caching reduces latency and decreases the load on CoreDNS pods.

Load Balancing: Distribute DNS query load across multiple CoreDNS instances. This can be achieved by configuring Kubernetes services and deployment strategies appropriately.

Resource Allocation: Allocate sufficient memory and CPU resources to CoreDNS pods, especially in large clusters with heavy DNS query traffic.

Tuning for High Availability and Performance: In high-demand environments, tune CoreDNS for high availability and performance. This includes configuring health checks, replicas, and autoscaling features.

Scaling CoreDNS

Horizontal Scaling: The most effective way to scale CoreDNS is horizontally. Increase the number of CoreDNS pods as the load increases. Kubernetes’ horizontal pod autoscaler can automate this process based on metrics like CPU or memory usage.

Cluster Size Consideration: The scaling strategy should consider the size of the Kubernetes cluster. Larger clusters with more services and pods will require more CoreDNS instances.

Autoscaling Configuration: Implement and configure autoscaling for CoreDNS deployments. This ensures that the number of CoreDNS pods adjusts dynamically in response to the current load.

Security: Safeguarding Kubernetes DNS Services

When it comes to Kubernetes and its components, such as CoreDNS, the significance of security is amplified due to the critical nature of DNS services in cluster operations. This section focuses on securing CoreDNS deployments, outlines best practices for DNS security in Kubernetes, and addresses common DNS-related security vulnerabilities.

Securing CoreDNS Deployment

Restricted Access Control: Implement strict access controls to the CoreDNS configuration (Corefile). Use Kubernetes RBAC (Role-Based Access Control) to control who can view or edit the ConfigMap for CoreDNS.

Using Network Policies: Employ Kubernetes network policies to restrict the traffic to CoreDNS pods, ensuring only legitimate requests from specific namespaces or pods are allowed.

Pod Security Policies: Implement Pod Security Policies (PSP) in Kubernetes to limit the potential attack surface. This includes restricting what CoreDNS pods can do and which resources they can access.

Secure Communication: Ensure all communications to and from CoreDNS are encrypted, particularly if integrating with external systems or cloud services for DNS resolution.

Regular Updates and Patching: Keep CoreDNS and Kubernetes regularly updated with the latest patches. Many security vulnerabilities are mitigated through routine updates.

DNS Query Monitoring and Analysis: Actively monitor DNS queries handled by CoreDNS. Analyzing query patterns can help in identifying potential DNS-based attacks like DNS amplification or spoofing.

Operational challenges - CoreDNS

Some of the common operational challenges associated with CoreDNS are highlighted as follows along with possible solutions.

DNS Resolution Failures: One of the most frequent issues is the failure of DNS resolution, where pods cannot resolve service names to IP addresses.

High Latency in DNS Queries: Occurs when DNS queries take an unusually long time to return, impacting the performance of applications.

Intermittent DNS Outages: Characterized by sporadic loss of DNS functionality, often due to resource constraints or configuration errors.

Incorrect DNS Data: Instances where DNS queries return incorrect or outdated information.

Following are the diagnostic tools and troubleshooting techniques:

Using kubectl for Initial Diagnosis: Start by checking the status of CoreDNS pods using kubectl get pods -n kube-system. Ensure that all CoreDNS pods are running and not frequently restarting.

Inspecting CoreDNS Logs: Use kubectl logs to inspect the logs of CoreDNS pods. Look for error messages or abnormal patterns that could indicate the root cause of the issue.

CoreDNS Metrics and Prometheus: If integrated with Prometheus, review the CoreDNS metrics for insights into query volumes, latencies, and error rates.

Corefile Configuration Check: Verify the Corefile configuration for errors or misconfigurations. Ensure that plugins are correctly configured and that there are no syntax errors.

Network Policy and Security Group Verification: Ensure that network policies or security groups are not inadvertently blocking traffic to CoreDNS pods.

Conclusion

This chapter provided a comprehensive overview of CoreDNS within Kubernetes, covering its architecture, deployment, and critical role in service discovery and DNS resolution. We delved into the nuances of CoreDNS Corefile configuration, how pods use CoreDNS for name resolution and the interaction between CoreDNS and other networking components. We also covered the performance and scalability aspects of CoreDNS. Finally, we explored potential issues and possible solutions regarding CoreDNS operations.

In the next chapter, we will explore Kubernetes networking, focusing on pod-to-pod and service-to-pod communications. It clarifies the unique IP address system for pods, and how Kubernetes Services facilitate traffic routing to pods. We will also discuss key concepts, such as Ingress Controllers, providing a comprehensive understanding of Kubernetes’ efficient and scalable networking capabilities.

Points to Remember

Default DNS Service: CoreDNS is the Default DNS Service in Kubernetes.

Plugin Architecture: CoreDNS features a flexible and extensible plugin architecture, allowing for customizable DNS services in Kubernetes.

Configuring CoreDNS Using ConfigMaps: CoreDNS is configured in Kubernetes clusters through ConfigMaps, enabling administrators to customize DNS settings.

Importance of DNS Security: Securing CoreDNS and its configurations is vital to protect against DNS-related vulnerabilities in Kubernetes environments.

Scalability of CoreDNS: CoreDNS can be scaled horizontally to manage increased DNS query loads in Kubernetes clusters.

Troubleshooting Common CoreDNS Issues: Understanding common issues like DNS resolution failures and knowing how to troubleshoot them is crucial for Kubernetes administrators.

CoreDNS and Kubernetes Networking: CoreDNS integrates tightly with Kubernetes networking, enabling efficient pod-to-pod and pod-to-service communication within the cluster.

Multiple Choice Questions

What is the primary role of CoreDNS in a Kubernetes cluster?

Load balancing

Service discovery and DNS resolution

Data encryption

Monitoring cluster performance

Which of the following is a key feature of CoreDNS’s architecture?

Monolithic design

Plugin architecture

Static configuration

In-memory data storage

In Kubernetes, what is CoreDNS commonly used for?

Managing container storage

Handling DNS queries within the cluster

Orchestrating container deployment

Automating code deployment

Which Kubernetes resource is used to configure CoreDNS?

Deployment

Service

ConfigMap

Pod

What is a common issue faced with CoreDNS in Kubernetes?

Image pull errors

DNS resolution failures

CPU overutilization

Ingress controller errors

Which of the following tools is essential for monitoring CoreDNS performance?

Docker

Prometheus

Git

Jenkins

In the context of CoreDNS, what does ‘DNS Security’ primarily focus on?

Enhancing website security

Protecting DNS deployments from vulnerabilities

Encrypting data at rest

User authentication

What is a key strategy for scaling CoreDNS in a Kubernetes environment?

Horizontal scaling

Vertical scaling

Decreasing pod replicas

Static resource allocation

Answers

b

b

b

c

b

b

b

a

Questions

How does DNS impact Kubernetes clusters?

How to balance flexibility and security in DNS configurations?

How to optimize CoreDNS performance in Kubernetes?

Keywords

CoreDNS

Service Discovery

DNS Resolution

Plugin Architecture

DNS Security

Performance Optimization

CHAPTER 13

Networking - Pod Service and Ingress

Introduction

In the last chapter, we explored CoreDNS, a crucial component for DNS resolution and service discovery in Kubernetes. It focused on CoreDNS’s architecture, its operational strategies, and how it revolutionizes service communication within Kubernetes clusters. The major discussions included CoreDNS’s plugin-based design for enhanced flexibility and customization, its integration with Kubernetes for efficient service discovery, and the operational challenges with practical solutions.

In this chapter, we will explore the Kubernetes networking model. Networking is a crucial component that enables applications to scale and interact efficiently. Understanding how networking functions between pods and between services and pods is fundamental for anyone working with Kubernetes. This chapter aims to demystify these concepts, providing a clear understanding of how Kubernetes networking operates.

Structure

In this chapter, we will cover the following topics:

Kubernetes Networking Model

Within Pod Communication (Multi Containers)

Pod-to-Pod Communication

Service-to-Pods Communication

External-to-Service Communication

Kubernetes Networking Model

The Kubernetes network model defines how various elements within a Kubernetes cluster communicate with each other. This model is unique and distinct from traditional networking approaches due to its focus on simplicity, transparency, and scalability to support containerized applications. Here are the key characteristics of the Kubernetes network model:

Within Pod communication (Multi Containers)

Containers within a Pod share the same network namespace, meaning they can communicate with each other using localhost. When a container, say C1, wants to communicate with another container, say C2, it sends network packets to the shared Pod IP address via the local eth0 interface. Since both containers share the same network namespace, the communication is direct and does not need to leave the Pod to reach the other container. The packets are not sent to the node’s network but are directly delivered to the destination container.

Pod-to-Pod Communication

Kubernetes provides flat networking space across pods. Every Pod gets its unique IP address, meaning no need for NAT (Network Address Translation) for pod-to-pod communication across the cluster. This flat network ensures that pods can communicate with each other directly, without requiring any special configuration like NAT.

Kubernetes employs Container Network Interface (CNI) plugins to interface with various networking solutions, offering flexibility and compatibility with different environments and requirements. CNI plugin assigns IP addresses to Pods. CNI plugin manages the network connectivity of pods and can implement additional networking features like network policies. We will explore more on CNI plugins in the next chapter.

Pod-to-Service Communication

Kubernetes Services provide an abstract way to expose applications running on a set of Pods. Services offer a single, stable IP address and DNS name entry point for connecting with pods, regardless of the internal pod structure or lifecycle.

Services manage load balancing and can be exposed to external networks, enabling communication with applications outside the Kubernetes cluster. Kube-apiserver assigns IP addresses to Services.

External-to-Service Communication

In Kubernetes, external-to-service communication is facilitated through specific types of services like NodePort, LoadBalancer, or Ingress. A NodePort service exposes a service on each Node’s IP at a static port, allowing external traffic to reach the service via any node’s IP and the NodePort. A LoadBalancer service integrates with cloud providers’ load balancers to route external traffic to the service, providing a public-facing IP address. Ingress, on the other hand, is a more sophisticated method that manages external access to the services in a cluster, typically HTTP/HTTPS, and can provide load balancing, SSL termination, and name-based virtual hosting. This architecture allows seamless and controlled access to cluster services from outside the Kubernetes cluster.

Understanding Within Pod communication (Multi Containers)

Containers within the same Pod in Kubernetes share a network namespace, allowing them to communicate directly via `localhost`. They use the Pod’s virtual ethernet device (eth0) for network interactions. The containers transmit data to each other using the local network without the need to route through the node’s network stack, ensuring efficient and low-latency communication. The virtual ethernet pair (veth) connects the Pod to the node’s network, but is not used for container-to-container communication within the same Pod (refer Figure 13.1).

In some kubernetes clusters, there may be a network bridge (cbr0 or docker0 or something similar) between veth0 and node eth0 based on which CNI plugin is used. This setup uses the CNI plugin, kindnet, a default CNI plugin as part of the KIND cluster setup, and it optimizes performance for inter-container communication.

Figure 13.1: Container-Container communication

To further illustrate this, let us deploy the following code snippet, as shown in Figure 13.2 in the KIND cluster.

Figure 13.2: Hello-net deployment

This kubernetes deployment (k8s-net) has two containers named c1 and c2. Both the containers use brainupgrade/hello:net image to launch a server program. The container c1 uses the port 8080 while the container c2 uses the port 8081 to run the server. Since the network is shared among containers within the same pod, ports can’t be the same. The following command deploys the aforementioned Kubernetes application:

kubectl apply -f https://raw.githubusercontent.com/brainupgrade-in/book-kubernetesadmin/main/chapter_13/01-within-pod-c2c-comm.yaml

Use the following command to know on which cluster node, the pod is running (Figure 13.3):

kubectl get pods -owide

Figure 13.3: Pod node allocation

To know the cluster nodes, you can use the following command (Figure 13.4):

kubectl get nodes -owide

Figure 13.4: Cluster nodes

Now, you can use curl command within each container to access the server running on another container (refer to Figure 13.5):

Figure 13.5: Containers on localhost

As you can see, both the containers responded with the same hostname. You can also verify if both the containers show the same host IP using ip route command (refer to Figure 13.6)

Figure 13.6: Containers route

Both containers share the POD ip which is 10.244.1.2 in this case, as shown in Figure 13.6.

Understanding Pod-to-Pod Communication - Same Node

As noted in the earlier section - Kubernetes Networking Model, Pod-to-Pod communication happens over IPs without NAT. Let us extend the earlier example, k8s-net, to understand this in greater detail. Scale the deployment using the following command and once all the replicas are running, you can obtain the nodes on which those are running too (refer to Figure 13.7):

kubectl scale --replicas 4 deploy k8s-net

Figure 13.7: Deployment Scaling - Pod allocation

Let us focus on the pods running on the node (net-worker2) now. Use curl to verify that the pods (k8s-net-c469fdbf8-z26v9 having IP 10.244.1.2 and k8s-net-c469fdbf8-k6jqm having IP 10.244.1.6) are able to communicate with each other (refer to Figure 13.8):

Figure 13.8: Pod-to-Pod communication - Same Node

Using ip route command on the node (net-worker2) and both the pods, we can link the communication between these two pods. Please refer to Figure 13.9 for a visual representation:

Figure 13.9: IP Routes and Virtual Ethernet Interfaces

When the following is executed, traffic to the pod (10.244.1.6) is routed via veth8d085d14.

kubectl exec k8s-net-c469fdbf8-z26v9 -c c1 -- curl -s 10.244.1.6:8080

And when the following is executed, traffic to the pod (10.244.1.2) is routed via vethfe97f0d0.

kubectl exec k8s-net-c469fdbf8-k6jqm -c c1 -- curl -s 10.244.1.2:8080

And both the veth interfaces are on the same host net-worker2 (172.18.0.2).

Figure 13.10 illustrates the communication path between two pods on the same node.

Figure 13.10: Pod to Pod - same node

Understanding Pod-to-Pod Communication - Different Nodes

If the pod being accessed is running on a different pod than the current one, then based on the routing rules set up on the host node, requests will be routed. For example, let us consider the pod k8s-net-c469fdbf8-z26v9 (10.244.1.2) running on net-worker2 (172.18.0.2) is trying to connect to the pod k8s-net-c469fdbf8-dcmvj (10.244.2.8) running on net-worker (172.18.0.3). In this case, the request will be routed via eth0 interface of node net-worker2 to the eth0 interface of node net-worker since both the nodes are part of the same subnet (172.18.0.0/16)

Figure 13.11: Routes on net-worker2

In essence, overall communication can be visualized in Figure 13.12:

Figure 13.12: Pod-to-Pod communication - different Node

Understanding Pod-to-Service Communication

Kubernetes services are a critical abstraction that provide a consistent way to expose applications running on a set of Pods as a network service. With services, Kubernetes enables loose coupling between microservices in your application by assigning a stable IP address across the cluster (known as ClusterIP) and DNS name to a group of Pods, thereby managing load balancing and service discovery efficiently. This allows for seamless scaling and updating of applications without disrupting the network communication within the Kubernetes cluster.

When a Kubernetes service is created, kube-proxy updates the iptables or IPVS rules to capture traffic destined for the service’s ClusterIP and port. kube-proxy itself does not directly forward traffic; instead, it configures the underlying network layer on each node to manage traffic routing. It does this primarily using Network Address Translation (NAT) rules. When kube-proxy operates in iptables mode, it sets up NAT rules in iptables, which the Linux kernel’s networking stack, then uses to redirect incoming traffic to the appropriate Pod IP addresses based on the service’s ClusterIP and port. This approach allows kube-proxy to leverage the existing network infrastructure of the node to efficiently handle service traffic routing and load balancing.

A pod in Kubernetes discovers a service’s ClusterIP through the cluster’s internal DNS service. When a service is created, it is automatically assigned a unique ClusterIP, and an associated DNS record is created. Pods query this DNS using the service name to obtain the ClusterIP for network communication.

Let us use the same deployment, k8s-net, and extend it to further illustrate pod-to-service communication. Use the following command to expose the deployment k8s-net:

kubectl expose deploy k8s-net --port 80 --target-port 8080

This creates a service named k8s-net which now can be used to access k8s-net pods. This service creation also results in Kube-proxy making changes to linux iptables enabling the traffic routing from service ClusterIP to its endpoints, that is, pods IPs. Kube-proxy makes iptables changes on all the cluster nodes. Please refer to the following snapshot:

Figure 13.13: Kube-proxy and iptables routing

Figure 13.13 depicts any request to k8s-net service gets one of its endpoints in random order and thereafter the request is routed as per the routing defined on the node.

If the kubernetes service is deleted, kube-proxy immediately deletes the entries it made in the iptables on all the cluster nodes.

Understanding External-to-Service Communication

External communication to a Kubernetes service is typically handled via NodePort, LoadBalancer, or Ingress services. NodePort exposes the service on a specific port of each node’s IP, while LoadBalancer uses cloud provider-specific load balancers to provide an external IP for the service. Ingress controllers, alternatively, manage external HTTP/HTTPS access, offering features like SSL termination and URL-based routing.

Via NodePort Service

When NodePort service is created, kube-proxy creates entries in iptables to forward traffic coming to NodePort to service and service to its endpoints. This means, one additional layer of routing created by kube-proxy is the service of type NodePort is defined.

To illustrate this further, let us create a NodePort service for k8s-net using the following commands.

kubectl create service nodeport k8s-net-nodeports --tcp=80:8080

Also, let us set the selector so that this new service of type NodePort can use the pods created by the deployment, k8s-net.

kubectl set selector svc k8s-net-nodeports app=k8s-net

All the entries created by kube-proxy for this service, k8s-net-nodeports, can be seen in the following snapshot (Figure 13.14):

Figure 13.14: Kube-proxy NAT rules for k8s-net-nodeports svc

Via LoadBalancer

In Kubernetes, a LoadBalancer service type automatically provisions an external load balancer (provided by the cloud platform) that directs traffic to the service’s NodePort. When a LoadBalancer service is used in Kubernetes, the external load balancer routes incoming traffic to the NodePort on the worker nodes. Each node then forwards this traffic to the appropriate backend pods, either directly or via kube-proxy managing the iptables or IPVS rules. This way, the NodePort acts as a bridge between the external load balancer and the internal ClusterIP service, enabling external traffic to reach the service.

Ingress-to-Service Communication

In Kubernetes, Ingress is a resource that manages external access to services within a cluster, typically for HTTP/HTTPS traffic. When a request hits the Ingress controller (a pod itself), it uses the rules defined in the Ingress resource to forward the request to the appropriate service. The Ingress controller looks at the host and path in the request and matches it against the Ingress rules to determine the destination service. The request is then forwarded to the service’s ClusterIP, which routes the traffic to one of the underlying pods based on its load-balancing algorithm. This setup allows efficient management of traffic routing, SSL termination, and name-based virtual hosting.

To illustrate this further, let us create an ingress resource that maps the requests coming to the host k8s-net.brainupgrade.in to the service k8s-net. Here is the ingress resource definition (Figure 13.15):

Figure 13.15: Ingress resource for k8s-net service

Apply this ingress resource using the following command:

kubectl apply -f https://raw.githubusercontent.com/brainupgrade-in/book-kubernetesadmin/main/chapter_13/02-k8s-net-ingress.yaml

This ingress definition requires the installation of an ingress controller. Since our cluster is KIND based and running locally, we can use the following bare metal ingress controller.

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/baremetal/deploy.yaml

Once the ingress controller gets running, you will also see that an ADDRESS (172.168.0.2) is allocated to the ingress resource that was created (refer to Figure 13.16):

Figure 13.16: Ingress k8s-net having ADDRESS

Now k8s-net service can be accessed via this load balancer IP (172.18.0.2) and ingress NodePort (31836 in this case). Refer to the following snapshot, wherein the application, k8s-net, is being accessed from outside the cluster using curl command:

curl 172.18.0.2:31836 -H ‘Host: k8s-net.brainupgrade.in’

Figure 13.17: Access k8s-net using load balancer IP

In the context of cloud-based Kubernetes cluster, Cloud Service creates a load balancer public URL and routes the traffic to ingress service on HTTP/HTTPS

Conclusion

This chapter provided a comprehensive overview of the Kubernetes networking model, emphasizing its uniqueness and efficiency in supporting containerized applications. We explored the nuances of within-pod communication, highlighting how containers within a pod share a network namespace, facilitating efficient and direct communication using localhost. The chapter also delved into pod-to-pod communication across the cluster, facilitated by a flat networking space and Container Network Interface (CNI) plugins, which assign unique IP addresses to pods, ensuring direct communication without NAT.

Additionally, we examined the critical role of Kubernetes services in abstracting pod access, offering stable IP addresses and DNS names for load balancing and service discovery, enhancing the scalability and resilience of applications. We also discussed how external communication to services is managed through NodePort, LoadBalancer, or Ingress, each serving different purposes and providing controlled access to cluster services from outside the Kubernetes environment.

In the upcoming chapter, we will delve into Container Network Interface (CNI) plugins, exploring their vital role in Kubernetes networking. We will examine how CNI plugins provide the networking layer for pods, enabling seamless communication within the cluster by assigning IP addresses and handling network routing. This chapter aims to demystify the functionality and diversity of CNI plugins, enhancing our understanding of Kubernetes’ network orchestration.

Points to Remember

Within Pod Communication: Containers within a pod share the same network namespace, allowing them to communicate using localhost without external routing.

Pod-to-Pod Communication: Kubernetes provides flat networking space, where each pod is assigned a unique IP, facilitating direct communication across the cluster without NAT.

Service-to-Pod Communication: Kubernetes Services offer an abstraction layer, providing a stable IP and DNS name for a group of pods and handling load balancing.

External to Service Communication: External traffic reaches cluster services through NodePort, LoadBalancer, or Ingress, each serving different external access needs.

Multiple Choice Questions

What do containers within the same Kubernetes pod use for communication?

Different IP addresses

localhost

External routing

Kubernetes employs which interface for pod networking?

Docker

Container Network Interface (CNI)

Pod Network Interface (PNI)

What is the primary role of Kubernetes Services?

Load balancing across nodes

Providing a stable IP and DNS name for a set of pods

Assigning unique IP addresses to pods

Which Kubernetes component is responsible for DNS-based service discovery?

kube-apiserver

kube-scheduler

CoreDNS

How does Kubernetes achieve pod-to-pod communication across the cluster?

Through a flat networking space without NAT

Via a centralized networking hub

By using dedicated physical network lines

What type of Kubernetes service exposes a service on each node’s IP at a static port?

ClusterIP

LoadBalancer

NodePort

Which Kubernetes resource manages external HTTP/HTTPS access to services within a cluster?

NodePort

Ingress

ExternalName

What does kube-proxy primarily use to manage service traffic routing?

Network Policies

NAT rules in iptables

Dedicated routing hardware

Which statement is true about external-to-service communication in Kubernetes?

Managed through NodePort, LoadBalancer, or Ingress

Only possible via Ingress

Requires a VPN

Answers

b

b

b

c

a

c

b

b

a

Questions

In what scenarios would a NodePort service be more advantageous than a LoadBalancer service in Kubernetes, and why?

What are the potential security implications of the flat networking space approach in Kubernetes, and how can they be mitigated?

How does the implementation of Ingress controllers in Kubernetes affect the management of external access to applications?

Key Terms

Services: Abstractions in Kubernetes that define a logical set of pods and a policy to access them, often providing a stable IP address and DNS name.

NodePort: A type of Kubernetes service that allows for external traffic to reach a service within a cluster by exposing a specific port on each node.

Ingress: A Kubernetes resource that manages external access to services within a cluster, typically for HTTP and HTTPS traffic, through rules defining traffic routing.

CoreDNS: A flexible, extensible DNS server that can be used in a Kubernetes cluster for service discovery, translating names to IP addresses.

Kube-proxy: A network proxy that runs on each node in a Kubernetes cluster, enabling the Kubernetes service abstraction by maintaining network rules.

ClusterIP: A type of service in Kubernetes that provides a unique IP address for accessing a set of pods, typically used for internal communications within the cluster.

LoadBalancer: A Kubernetes service type that provides external network access to a set of pods, often by integrating with cloud providers’ load balancers.

CHAPTER 14

Kubernetes CNI

Introduction

In the previous chapter, we explored Kubernetes networking in depth, covering various levels of communication: container-to-container within the same pod, pod-to-pod, service-to-pod, and external access to services. We understood how containers within a single pod share an IP address and can communicate through `localhost`, highlighting the internal networking model of a pod. We also covered inter-pod communication, services discovery and routing to pods, thus providing a stable interface to pod collections. Finally, we experimented with external communication to services using NodePort, LoadBalancer, and Ingress.

In this chapter, we will delve into Kubernetes’ Container Network Interface (CNI), detailing its role in integrating various network plugins for secure, isolated container communication. We will also explore popular CNI plugins such as Calico, Flannel, and Weave Net, each offering unique features for network management. The chapter will also cover the selection criteria, installation, and configuration of these plugins, emphasizing their significance in enforcing network policies and ensuring efficient pod networking within Kubernetes clusters.

Examples referred to in this chapter are run in the multi-nodes cluster that is set up using the Kind method, with cluster.yaml provided as an input configuration, as explained in Chapter 2: Installing Kubernetes.

Structure

In this chapter, the following topics will be covered:

Overview - Kubernetes CNI Model

CNI Plugins

Selection Criteria

CNI Plugin Installation - Calico

Kubelet and CNI Plugin Discovery

Pod Networking Setup

Calico CNI Plugin

Network Policy

Network Policy Enforcement by Calico

Flannel CNI Plugin

Weave Net CNI Plugin

Overview: Kubernetes CNI Model

The Kubernetes CNI (Container Network Interface) model is a framework for integrating container networking solutions. It allows Kubernetes to leverage various network plugins for container communication. Under this model, each pod gets its network interface and IP address, ensuring network isolation and security.

The CNI model defines a set of operations for network plugins, like creating, deleting, and querying network interfaces. It’s plugin-driven, enabling users to choose the most suitable networking solution for their cluster. Plugins can provide functionalities such as overlay networks, network policies, and load balancing. Kubernetes invokes the configured CNI plugin to set up networking for each pod. This setup simplifies network management in Kubernetes and provides flexibility. The CNI model supports different networking topologies, adapting to diverse infrastructure needs. It’s essential for Kubernetes’ scalable and dynamic networking capabilities, catering to complex, distributed applications.

To facilitate CNI plugins development and operations, a detailed set of rules are defined in the form of CNI (Container Network Interface) specification. The explanation is provided as follows:

Standardized Interface: For container networking in Linux environments.

Network Creation and Deletion: Defines how networks are set up and torn down.

IP Management: Outlines IP address allocation for containers.

Configuration: Specifies network configuration in JSON format.

It serves as a guideline for creating compatible network plugins in containerized environments.

CNI Plugins

As of this writing, there are many CNI plugins available. A few of the leading CNI plugins are listed as follows:

Calico: Known for its network policy enforcement and performing

Flannel: Popular for its simplicity and ease of setup

Weave net: Offers robust features and simplicity

Cilium: Uses BPF for advanced security and networking capabilities

Canal: Comines Flannel and Calico, providing features of both

You can find many third party supported plugins here: https://github.com/containernetworking/cni

Selection Criteria

CNI plugin selection criteria in Kubernetes often include:

Network Requirements: Based on specific network topology, performance, and isolation needs.

Compatibility: With the Kubernetes version and cluster configuration.

Feature Set: Like network policy support, load balancing, and logging.

Community Support and Stability: Well-supported, widely-used plugins offer reliability.

Ease of Use: Simplicity in installation and maintenance.

These criteria guide in choosing the right CNI plugin for a Kubernetes environment.

Installing CNI Plugin - Calico

Installation steps are well documented by CNI Plugin providers and generally published on their main website. To go through a sample installation, let us consider Calico as a CNI plugin that we want to install in our KIND cluster.

First, create a kind cluster by disabling the CNI. Delete any previously created KIND cluster. Download the cluster config yaml file (cluster-disable-cni.yaml) from here: https://raw.githubusercontent.com/brainupgrade-in/book-kubernetesadmin/main/chapter_14/cluster-disable-cni.yaml and run the following command:

kind create cluster --name cni --config cluster-disable-cni.yaml

Once cluster installation is over, get the nodes status using the following command and you will notice that all the cluster nodes have the status NotReady (refer to Figure 14.1).

kubectl get nodes

Figure 14.1: Nodes - NotReady

Run the following command and you will know the reason:

kubectl get nodes cni-control-plane -ojsonpath=’{.status.conditions[?(@.type==”Ready”)]}’

And the output would be similar as given here:

{“lastHeartbeatTime”:”2024-01-15T20:40:59Z”,”lastTransitionTime”:”2024-01-15T20:25:36Z”,”message”:”container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized”,”reason”:”KubeletNotReady”,”status”:”False”,”type”:”Ready”}

Kubelet does various checks on the cluster nodes and one of the critical checks is to ensure the networking plugin is ready. And from the preceding logs, you can clearly see that the network (CNI) plugin is not ready.

Now let us proceed with the Calico CNI plugin installation. Calico Plugin provided by Tigera documented installation steps are given here: https://docs.tigera.io/calico/latest/getting-started/kubernetes/kind.

And based on the steps, install Calico operator and custom resources using the following commands:

kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.27.0/manifests/tigera-operator.yaml

kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.26.4/manifests/custom-resources.yaml

Verify the CNI plugin installation using the following command:

kubectl get pods -l k8s-app=calico-node -A

If you see the output showing calico pods in Running status, then verify the cluster nodes status again using the command kubectl get nodes and you will notice that now the cluster nodes are in Ready status. Refer to the following screenshot (Figure 14.2):

Figure 14.2: Cluster Nodes - Ready - Calico CNI

Kubelet and CNI Plugin Discovery

When starting, kubelet is configured with specific command-line arguments that dictate its behavior regarding networking. The --network-plugin=cni flag tells kubelet to use the CNI for networking. It also includes paths for the CNI configuration directory (for example, --cni-conf-dir=/etc/cni/net.d) and the CNI binary directory (for example., --cni-bin-dir=/opt/cni/bin).

Kubelet checks the CNI configuration directory for network configuration files. These files are typically in JSON format and specify how the pod networking should be handled, including which CNI plugin to use. If there are multiple configuration files, kubelet selects one based on file naming order (alphabetically/numerically). The selected configuration file specifies the CNI plugin to be used, often indicated by a type field in the JSON.

With the plugin identified, kubelet then looks for the corresponding executable in the CNI binary directory. The plugin binary is a standalone executable that kubelet will run to set up or tear down the pod network.

Pod Networking Setup

The interaction between kubelet and the CNI plugin, in the context of how a pod gets its IP address in Kubernetes, involves several key steps:

Pod Scheduling:

A pod is created and scheduled to run on a specific node in the Kubernetes cluster.

Kubelet Reacts to Pod Scheduling:

Kubelet on the designated node detects the new pod assignment.

It initiates the process to start the pod, part of which involves setting up the network.

Kubelet Invokes CNI Plugin:

Kubelet looks for a CNI configuration file in the specified directory (for example, /etc/cni/net.d). This file defines which CNI plugin to use and its configuration.

Kubelet executes the CNI plugin binary (located in a directory like /opt/cni/bin), passing the network configuration from the file.

CNI Plugin Network Setup:

The CNI plugin sets up the network for the pod. This typically involves creating a network namespace, creating veth pairs (virtual ethernet devices), and attaching one end to the pod’s network namespace.

The CNI plugin is responsible for IP address management (IPAM). It allocates an IP address for the pod from the configured range.

IP Address Assignment:

The CNI plugin assigns the selected IP address to the pod’s network interface.

This IP address allows the pod to communicate with other pods and services in the cluster.

Routing and DNS Configuration:

The CNI plugin sets up necessary routes so the pod can communicate with the Kubernetes network.

It also configures DNS settings for the pod, allowing it to resolve cluster service names.

Completion and Handoff:

After the network setup, the CNI plugin returns control to kubelet, signaling completion along with any relevant network information (like the assigned IP address).

Kubelet then continues the pod startup process, starting containers within the pod.

Pod Networking Ready:

With the network set up and containers running, the pod is now ready and operational with its own unique IP address.

This IP address is used for all network communications to and from the pod.

This process illustrates how kubelet coordinates with the CNI plugin to ensure each pod in the Kubernetes cluster has the necessary network setup, including a unique IP address for communication within the cluster.

Calico CNI Plugin

The Calico CNI plugin is a powerful and flexible networking solution for Kubernetes. It provides high-performance, scalable, and secure networking capabilities. Calico supports advanced network policies for granular security control. It utilizes standard Linux networking and routing, optionally leveraging eBPF for enhanced performance. Calico is widely adopted for its ease of use and comprehensive feature set in diverse Kubernetes environments.

Calico’s key architectural components include:

Data Plane: Uses Linux IP routing, optionally accelerated with eBPF, for packet forwarding.

Control Plane: Orchestrates network policies and manages IP addresses for pods.

BGP (Border Gateway Protocol): For efficient routing and connectivity between nodes.

Felix: Primary agent on each node, responsible for implementing network policies and routes.

Typha: Optimizes Felix’s access to the Kubernetes API, beneficial in large clusters.

This architecture ensures scalable, secure, and efficient networking.

Main features of Calico for Kubernetes networking include:

Network Policy Enforcement: Advanced security with fine-grained access controls.

High Performance: Efficient routing using IP-in-IP or VXLAN.

Scalability: Suitable for large-scale and complex networks.

Cross-Subnet Networking: Supports communication across different subnets.

IPv4 and IPv6 Support: Dual-stack compatibility for modern networking needs.

Calico is known for its robust security and performance capabilities.

Network Policy

A Network policy in Kubernetes is a specification of how groups of pods are allowed to communicate with each other and other network endpoints. It allows you to define rules for ingress (incoming traffic) and egress (outgoing traffic) to and from pods. Network policies are implemented using labels to select pods and define rules which specify what traffic is allowed to and from these pods. They provide an additional layer of security in Kubernetes by helping to restrict the flow of traffic, which is essential in enforcing a zero-trust networking environment. It’s important to note that Network policies are enforced by the network plugin in use (such as Calico), and not all network plugins support them.

We will explore more on how to define Network Policy rules in Chapter 15: Kubernetes Security.

Network Policy Enforcement by Calico

Calico’s policy enforcement mechanism is integral to securing pod traffic in a Kubernetes cluster, allowing administrators to define fine-grained rules for how pods can communicate and ensuring these rules are consistently enforced at the network level. Following are the steps taken by Calico to enforce network policies in Kubernetes:

Policy Application:

When a pod is created or updated in Kubernetes, Calico checks for any applicable network policies based on the pod’s labels.

These policies dictate how traffic should be allowed or blocked to and from the pod.

Programming the Data Plane:

Calico enforces these policies at the data plane level.

This is primarily achieved by programming iptables rules (or using eBPF, if enabled) on the host node where the pod is running.

The rules are configured to match the traffic patterns (source/destination IPs, ports, protocols) as specified in the network policies.

Stateful Policy Enforcement:

Calico supports stateful policy enforcement. This means it can track the state of network connections (like TCP connections) and apply rules based on this state.

For instance, it can allow return traffic for a connection that was initiated from within a pod, even if the inbound rules are restrictive.

Dynamic Updates:

As network policies are added, removed, or updated in Kubernetes, Calico dynamically updates the iptables/eBPF rules.

This ensures that the policy enforcement is always in sync with the current set of policies defined in the cluster.

Isolation and Segmentation:

By enforcing these policies, Calico provides network isolation and segmentation.

Pods can be isolated from each other, and distinct segments of a cluster can have different policy rules, enhancing security and reducing the risk of unauthorized access.

Logging and Visibility:

Calico can log attempts to establish connections that are blocked or allowed by network policies.

This provides visibility into network traffic patterns and potential security issues.

It is essential to have a good understanding of at least one CNI plugin even though the CKA certification exam may not ask any question on the CNI plugin itself. In the rest of the sections, let us understand a couple of more CNI plugins at a high level.

Flannel CNI Plugin

The Flannel CNI plugin is a simple and easy-to-use network fabric for Kubernetes. It creates an overlay network that connects Kubernetes pods across different nodes. Flannel uses a flat layer-3 network model that assigns each pod a unique IP address, enhancing simplicity in network management. It’s known for its straightforward setup and minimal configuration, making it a popular choice for smaller or simpler Kubernetes deployments. Flannel supports various backends such as VXLAN and UDP, providing flexibility in how data packets are transported across the network.

Flannel CNI Plugin’s main features include:

Overlay Networking: Creates a virtual layer-3 network for pods, ensuring seamless inter-pod communication across different nodes.

Simple Configuration: Known for its ease of setup and minimal operational configuration, making it user-friendly, especially for new Kubernetes users.

Multiple Backend Support: Supports various backends such as VXLAN, UDP, and host-gateway, offering flexibility in network traffic handling.

Network Isolation: Provides basic network isolation between pods and physical network infrastructure, enhancing security.

Integration with Kubernetes: Designed to integrate smoothly with Kubernetes, automatically managing and assigning subnet leases to nodes with the etcd datastore.

Flannel is popular in the Kubernetes community for its simplicity and effectiveness in providing essential networking capabilities.

Typical use cases for the Flannel CNI plugin in Kubernetes environments include:

Simple and Small Clusters: Ideal for small-scale or test Kubernetes clusters due to its simplicity and ease of setup.

Development and Testing Environments: Often used in development environments where advanced networking features are not required.

CI/CD Pipelines: Suitable for continuous integration and continuous deployment pipelines where quick and easy network setup is beneficial.

Learning and Experimentation: Frequently chosen for educational purposes or by beginners learning Kubernetes networking concepts.

Basic Network Requirements: Well-suited for scenarios where complex networking features like network policies or advanced routing are not needed.

Flannel’s straightforward and minimal configuration approach makes it a good choice for these scenarios.

Weave Net CNI Plugin

The Weave Net CNI plugin is a resilient and easy-to-use networking solution for Kubernetes. It establishes a robust overlay network for containers and supports various topologies. Weave Net automatically manages network configuration and simplifies tasks like address assignment and name resolution. It offers built-in encryption for secure communications across clusters. Additionally, Weave Net is known for its ability to maintain connectivity even in dynamic and changing environments, making it a reliable choice for complex deployments.

The main features of the Weave Net CNI plugin are:

Overlay Networking: Creates a virtual network that connects Docker containers across multiple hosts, allowing for seamless inter-container communication.

Network Encryption: Offers built-in encryption options for securing data in transit across the cluster, enhancing security.

Ease of Use: Simple to set up and manage, with minimal configuration required, making it accessible for users of various expertise levels.

Fault Tolerance: Designed for high availability and fault tolerance, ensuring reliable networking even in complex scenarios.

Support for Network Policies: Implements Kubernetes network policies, allowing for fine-grained control over container communication.

Typical use cases for the Weave Net CNI plugin include:

Multi-Host Kubernetes Clusters: Ideal for connecting containers across different hosts in a Kubernetes cluster, especially in environments where cross-host communication is critical.

Secure Environments: Suitable for deployments requiring encrypted network communications, as Weave Net provides built-in encryption features.

Complex and Dynamic Environments: Its ability to automatically adapt to network changes makes it a good fit for dynamic environments with frequently changing infrastructure.

High Availability Setups: Given its fault tolerance, Weave Net is well-suited for high-availability Kubernetes deployments where network reliability is paramount.

Development and Testing: Its ease of setup and management makes Weave Net a convenient choice for development and testing environments in Kubernetes

Conclusion

This chapter provided a comprehensive overview of Kubernetes CNI, essential for integrating varied network plugins and ensuring secure, isolated container communication. It highlights the functionality, selection, and setup of key CNI plugins such as Calico, Flannel, and Weave Net. The chapter underscores the interaction between kubelet and CNI plugins for efficient pod networking. This chapter is pivotal for understanding the complexities and capabilities of networking in Kubernetes environments.

The next chapter delves into the critical aspects of Kubernetes security, focusing on Role-Based Access Control (RBAC), Network Policies, Roles, Service Accounts, and User Groups. It emphasizes the configuration and management of access controls and permissions to ensure secure operations within Kubernetes. The chapter explains how Network Policies regulate pod-to-pod communication and the significance of Roles and Service Accounts in defining and assigning specific privileges. Additionally, it covers the management of User Groups for organized and efficient access control in Kubernetes environments.

In this chapter, we will explore the fundamental aspects of security within a Kubernetes environment.

Points to Remember

Kubernetes CNI Framework: Understand the Container Network Interface (CNI) as a pivotal framework in Kubernetes for integrating various network plugins, ensuring secure and isolated container communication.

Popular CNI Plugins: Familiarize with key CNI plugins such as Calico, Flannel, and Weave Net, each offering unique features for network policy enforcement, simplicity, and robust networking.

Selection and Installation: Recognize the importance of selecting the right CNI plugin based on network requirements, compatibility, and ease of use, and comprehend the installation and configuration process.

Kubelet and CNI Interaction: Grasp the role of kubelet in CNI plugin discovery and the critical steps involved in setting up pod networking within a Kubernetes cluster.

Multiple Choice Questions

What is the primary function of Kubernetes’ Container Network Interface (CNI)?

Load balancing

Network plugin integration

Automatic scaling

Storage management

Which of the following is NOT a CNI plugin mentioned in this chapter?

Calico

Flannel

Docker Swarm

Weave Net

What is a key consideration when selecting a CNI plugin for Kubernetes?

Programming language used

Network requirements and compatibility

Color of the plugin logo

Which CNI plugin is known for its network policy enforcement capabilities?

Calico

Flannel

Weave Net

In the context of Kubernetes CNI, what role does kubelet play?

It deploys containers

It manages database connections

It coordinates with the CNI plugin for pod networking setup

Which CNI plugin is highlighted for its simplicity and ease of use, especially in small-scale or simpler Kubernetes deployments?

Calico

Flannel

Cilium

What does a Network Policy in Kubernetes primarily control?

Pod communication rules

CPU and memory allocation

Storage access permissions

Which feature is NOT typically provided by a Kubernetes CNI plugin?

Network isolation

Encryption of data in transit

Automatic code deployment

Answers

b

c

b

a

c

b

a

c

Questions

What future developments or improvements could be made to the Kubernetes CNI ecosystem to further enhance container networking capabilities and security?

How does the integration of CNI plugins with kubelet affect the overall efficiency and reliability of a Kubernetes cluster’s networking?

How do different CNI plugins like Calico, Flannel, and Weave Net compare in terms of performance, scalability, and ease of use in various Kubernetes environments?

Key Terms

Kubernetes CNI (Container Network Interface): A standard framework used in Kubernetes to provide networking for pods by integrating various network plugins.

Network Plugins: Software components that integrate with Kubernetes CNI to provide network connectivity and isolation for pods.

Calico: A popular CNI plugin known for its advanced network policy enforcement capabilities and high-performance networking features.

Flannel: A CNI plugin recognized for its simplicity and ease of use, often used in smaller or less complex Kubernetes deployments.

Weave Net: Another CNI plugin, known for its robust features and ease of use, providing a resilient networking solution for Kubernetes.

Kubelet: A Kubernetes component that interacts with CNI plugins to set up pod networking, ensuring each pod has a unique IP address and proper network connectivity.

Network Policies: Specifications in Kubernetes that define how groups of pods are allowed to communicate with each other and other network endpoints.

CHAPTER 15

Kubernetes Security

Introduction

The previous chapter provided an in-depth overview of Kubernetes’ Container Network Interface (CNI), detailing its role in integrating various network plugins for secure, isolated container communication. It explored popular CNI plugins such as Calico, Flannel, and Weave Net, focusing on their installation, configuration, and network policy enforcement. The chapter also highlighted the interaction between kubelet and CNI plugins in setting up pod networking.

In this chapter, we will explore the fundamental aspects of security within a Kubernetes environment. We will cover critical components like authentication, authorization, network policies, and best practices for securing cluster components, ensuring a comprehensive understanding of maintaining a secure Kubernetes ecosystem.

Structure

In this chapter, we will cover the following topics:

Overview - Kubernetes Security Model

API Server - Central Access Point

Authentication and Authorization

Admission Control

Pod Security Standards

RBAC - Role Based Access Control

Network Policies

Securing Control Plane

Audit Logging

Overview - Kubernetes Security Model

The Kubernetes Security Model is a comprehensive framework designed to secure a cluster’s infrastructure and its applications. It employs a multi-layered approach, integrating several mechanisms to ensure robust defense at various levels.

Authentication: The first line of defense involves confirming the identity of users or processes. Kubernetes supports multiple authentication methods like client certificates, bearer tokens, basic authentication, and external identity providers (for example, OpenID Connect).

Authorization: Post authentication, authorization determines what an authenticated user can do. Kubernetes uses Role-Based Access Control (RBAC), Attribute-Based Access Control (ABAC), Node authorization, and Webhook mode for fine-grained access control.

Admission Control: Admission controllers are plugins that intercept requests to the Kubernetes API server post-authentication and authorization. They modify or reject requests, enforcing policy compliance and ensuring that only valid resources are created or modified.

Network Policies: These define how pods communicate with each other and with other network endpoints, crucial for creating isolated environments within the same cluster and reducing attack surfaces.

TLS for Secure Communication: Kubernetes heavily relies on TLS encryption to secure all communication paths within the cluster. This includes communication between nodes, between pods, and between pods and the API server, safeguarding data in transit.

Secrets Management: Kubernetes stores and manages sensitive information such as passwords and tokens using Secrets. It ensures that sensitive data is not stored in plaintext in pod specifications or container images. In Chapter 9: Config Apps for Production Deployment, we covered this in detail.

Role of Etcd: As the primary data store for Kubernetes, securing etcd is critical. Access to etcd is tightly controlled and often encrypted, providing a secure storage mechanism for the cluster state and configuration. In Chapter 10: Cluster Database-Backup and Restore, we covered details about ETCD.

Audit Logging: Audit logs provide a chronological record of events, capturing detailed information about requests made to the Kubernetes API. This is vital for troubleshooting, monitoring, and forensic analysis in case of security incidents.

This security model enables Kubernetes to provide a secure environment for managing containerized applications, ensuring that the cluster’s operational integrity is maintained and that applications are safeguarded against unauthorized access or manipulation.

API Server - Central Access Point

The API Server in Kubernetes plays a pivotal role in the overall security of the Kubernetes cluster. It serves as the central control plane component, managing interactions between different parts of the cluster and external users or services. The API Server is the primary interface for all operations within the Kubernetes cluster. All internal and external requests to modify or view the cluster state go through the API Server. This centralized approach allows for effective monitoring and control of access to the cluster. Hence the API Server is crucial in the context of Kubernetes Security.

Authentication and Authorization

The API Server is responsible for authenticating users and services before they interact with the Kubernetes cluster. After authentication, it also performs authorization checks to ensure that the authenticated entity has the necessary permissions to perform the requested actions. This ensures that only authorized entities can access or modify cluster resources.

API Server Authentication Mechanisms:

Client Certificates: Enable and configure client certificate authentication by setting the --client-ca-file flag to point to your CA certificate. This allows the API Server to validate client certificates signed by your CA.

Bearer Tokens: Use --service-account-key-file to specify the public key for validating JWT tokens used by service accounts.

Basic Authentication: While not recommended for production, basic authentication can be enabled using a static password file specified with --basic-auth-file.

OIDC: Integrate with an OpenID Connect provider using flags like --oidc-issuer-url, --oidc-client-id, --oidc-username-claim, and so on.

API Server Authorization Mechanisms:

RBAC: Enable Role-Based Access Control (RBAC) using --authorization-mode=RBAC. This is the recommended authorization mode in Kubernetes and explained in detail in later sections of this chapter.

Node Authorization: Use --authorization-mode=Node for authorizing API requests made by kubelet.

ABAC: Attribute-Based Access Control (ABAC) can be enabled with --authorization-mode=ABAC and a policy file.

Webhook: Externalize authorization checks to a webhook using --authorization-mode=Webhook.

Admission Control

Admission controllers in Kubernetes are plugins that intercept requests to the Kubernetes API server prior to persistence of the object, but after the request is authenticated and authorized. They serve as a powerful mechanism to enforce governance and policy, validate or modify requests, and ensure that the cluster state complies with certain rules. There are two types of admission controllers, including:

Validating Admission Controllers: These controllers validate the requests and ensure they meet certain conditions before allowing them to proceed. If a request does not meet the necessary criteria, it is rejected, and an error is returned to the end-user. Examples include PodSecurity, LimitRanger, and ResourceQuota.

Mutating Admission Controllers: These controllers can modify requests before they are processed, often adding or altering default values. They can alter the object in the request before it gets persisted. For example, the DefaultStorageClass admission controller automatically assigns a default storage class to PersistentVolumeClaim objects that don’t request any specific class.

Admission controllers are integral to the API server and can be enabled or disabled as per the cluster’s requirements by using the flag --enable-admission-plugins. They play a crucial role in maintaining the security and integrity of the cluster by applying specific policies and rules. Some admission controllers, including NamespaceLifecycle, are enabled by default, while others must be explicitly enabled. Managing admission controllers effectively is key to ensuring a stable and secure Kubernetes environment.

Pod Security Standards

Pod Security Standards in Kubernetes are a set of predefined security profiles that define different levels of security requirements for pods. These standards are part of Kubernetes’ Pod Security Admission (PSA) control, which aims to simplify and standardize how security settings are applied to pods. The Pod Security Standards provide a more streamlined and user-friendly approach compared to the now-deprecated PodSecurityPolicies.

The Pod Security Standards consist of three distinct profiles, which are as follows:

Privileged:

This is the least restrictive profile. It allows for the greatest flexibility and is intended for pods that need a high level of privileges.

When using this profile, pods are not subject to any restrictions.

Baseline (Default):

This profile is designed to be broadly applicable to a wide range of workloads while providing a good level of security against common vulnerabilities.

It’s more restrictive than the Privileged profile but still provides enough permissions for most applications to run without modification.

It prevents pods from performing a set of actions that are known to be high risk, such as running as root or accessing host resources.

Restricted:

The most restrictive profile, the Restricted profile applies a comprehensive set of security best practices.

It’s intended for workloads that handle sensitive data and require the highest level of security.

This profile significantly restricts what a pod can do, ensuring that it operates in the most secure and least privileged mode possible.

These profiles are applied at the namespace level via labels, allowing cluster administrators to enforce different standards for different sets of workloads. By providing a more approachable way to apply security controls, Pod Security Standards aim to improve the overall security posture of Kubernetes clusters.

In Kubernetes Pod Security Standards, there are three modes available:

Enforce Mode: Actively enforces the Pod Security Standard, blocking non-compliant pods from being created or updated.

Audit Mode: Allows non-compliant pods but records violations in the cluster’s audit logs, useful for evaluating the impact of policy changes.

Warn Mode: Permits non-compliant pods but emits warnings to users during pod creation or update, providing immediate feedback on compliance issues.

Pod Security Standards are applied at the namespace level. Label your namespace with the appropriate level of the standard you want to enforce. For example, to apply the Baseline standard to a namespace named example-namespace, you would label it as follows:

kubectl label namespace example-namespace pod-security.kubernetes.io/enforce=baseline

RBAC - Role Based Access Control

RBAC (Role-Based Access Control) in Kubernetes is a method for regulating access to kubernetes resources based on the roles of individual users within an organization. RBAC follows the principle of least privilege, ensuring users have only the access necessary for their role. It involves the following:

Roles and ClusterRoles: Define permissions for resources within a namespace (Roles) or across the entire cluster (ClusterRoles).

RoleBindings and ClusterRoleBindings: Assign roles to user accounts, service accounts, or groups within a specific namespace (RoleBindings) or across the entire cluster (ClusterRoleBindings).

Example - Access to Dev Teams

Let us take an example wherein using RBAC, we want to provide only deploy and manage access to payments related applications in the payments namespace. Following are the steps to implementation.

Create User Credentials:

Create SSL key and CSR (Certificate Signing Request) using following openssl commands.

openssl genrsa -out payments-developer.key 2048

openssl req -new -key payments-developer.key -out payments-developer.csr -subj “/CN=payments-developer”

Obtain cluster signed certificate for the user:

Download Kubernetes CSR - CertificateSigningRequest yaml and add base64 data of payments-developer.csr using following commands.

wget -q https://raw.githubusercontent.com/brainupgrade-in/book-kubernetesadmin/main/chapter_15/01-payments-developer-csr.yaml

csrbase64=$(cat payments-developer.csr | base64 | tr -d ‘\n’)

sed -i “s/PLACEHOLDER/$csrbase64/g” 01-payments-developer-csr.yaml

Raise certificate request, approve the same and download client certificate file using following commands.

kubectl apply -f 01-payments-developer-csr.yaml

kubectl certificate approve payments-developer

kubectl get csr payments-developer -o jsonpath=’{.status.certificate}’ | base64 -d > payments-developer.crt

Setup cluster access

Setup access details for the client (payments-developer) in the kubeconfig file to access the cluster. The file, kubeconfig, is a configuration file used by kubectl and other Kubernetes clients to define cluster connections and authentication details. It allows users to organize information about clusters, users, namespaces, and authentication mechanisms. This file is generally kept inside the <USER-HOME>/.kube folder

kubectl config set-credentials payments-developer --embed-certs=true --client-certificate=payments-developer.crt --client-key=payments-developer.key

kubectl config set-context payments-developer-context --cluster=kind-cni --namespace=payments --user=payments-developer

Role and RoleBinding

Now, let us set up the role with appropriate privileges and assign it to this user. To create the role, use the following command. The flag --verb contains the allowed actions on the specified resources denoted using flag --resource

kubectl create role payments-developer --namespace=payments --verb=get,list,watch,create,update,patch,delete --resource=deployments,replicasets,pods,services

Bind this role to the user using the following command.

kubectl create rolebinding payments-developer-binding --namespace=payments --role=payments-developer --user=payments-developer

You can verify if the preceding configuration is working as intended using the following command.

kubectl auth can-i get pods -n payments --as payments-developer

If the output is yes, then switch to this payments-developer user’s context using the following command and proceed with deploying applications in the payments namespace.

kubectl config use-context payments-developer-context

kubectl config use-context payments-developer-context (to switch back to the admin context)

Example - Jenkins CICD

For applications like Jenkins, service accounts are used while binding roles/clusterroles. For the sake of illustration, let us say we want to use Jenkins for building and deploying the application in any namespace in the same cluster. Use the following steps to make it happen:

Create namespace and service account.

kubectl create ns jenkins

kubectl create serviceaccount jenkins --namespace=jenkins

Create a cluster role and bind it to allow Jenkins to deploy apps in any namespace.

kubectl create clusterrole jenkins-deployer --verb=create,delete,get,list,patch,update,watch --resource=deployments,services,pods,pods/log

kubectl create clusterrolebinding jenkins-deployer-binding --clusterrole=jenkins-deployer --serviceaccount=jenkins:jenkins

Create jenkins deployment and map it to the service using following commands:

kubectl create deploy jenkins -n jenkins --image jenkins=brainupgrade/jenkins:2.414.1x3

kubectl create svc nodeport jenkins -n jenkins --tcp=80:8080

Set serviceaccount jenkins to the Kubernetes deployment named jenkins.

kubectl set sa deploy jenkins jenkins -n jenkins

And here you go, a working CICD Jenkins app that can build and deploy kubernetes app in any namespace.

Network Policy

A network policy in Kubernetes is a specification that defines how pods can communicate with each other and with other network endpoints. It allows users to control the flow of traffic at the IP address or port level based on a set of rules, effectively enabling a form of network segmentation and isolation for pods. This is crucial for enhancing the security and integrity of containerized applications within a Kubernetes cluster, as it helps in implementing least privilege networking access. Network policies are applied at the namespace level and are enforced by the network plugin in use (the cluster must be configured to use a network plugin that supports Network Policy).

Example - HR and Payroll apps

Network policies in Kubernetes work by defining rules that govern ingress (incoming) and egress (outgoing) traffic to and from pods in a namespace. They enable you to specify which pods can communicate with each other, and which external entities they can interact with. Let’s illustrate this with an example where an HR application is deployed in the hr namespace and a Payroll application is deployed in the payroll namespace.

Scenario:

The HR application should be able to receive traffic from the Payroll application but should not initiate communication with it.

The Payroll application needs to send data to the HR application but should not accept any incoming connections from other services.

Creating Network Policies

Network Policy for HR Application: This policy allows ingress traffic from the Payroll application to the HR application. This policy selects all pods in the hr namespace and allows ingress traffic from pods in the payroll namespace (Figure 15.1).

Figure 15.1: Network Policy - HR App

Network Policy for Payroll Application: This policy allows egress traffic to the HR application but restricts ingress traffic. This policy selects all pods in the payroll namespace, allows egress traffic to the hr namespace, and denies all ingress traffic (Figure 15.2).

Figure 15.2: Network Policy - Payroll application

Create namespaces and apply the Network Policies using following commands:

kubectl create ns hr

kubectl create ns payroll

kubectl apply -f https://raw.githubusercontent.com/brainupgrade-in/book-kubernetesadmin/main/chapter_15/04-hr-network-policy.yaml

kubectl apply -f https://raw.githubusercontent.com/brainupgrade-in/book-kubernetesadmin/main/chapter_15/05-payroll-network-policy.yaml

Outcome:

The HR application can only receive traffic from the Payroll application but cannot initiate traffic to it.

The Payroll application can initiate traffic to the HR application but cannot receive traffic from other applications.

This example demonstrates how network policies can be effectively used to control traffic flow between different applications in separate namespaces, enhancing the security and isolation in a Kubernetes environment.

Testing Network Policies

To validate the effectiveness of the network policies implemented for the HR and Payroll applications, we can deploy simple applications in the hr and payroll namespaces and then test connectivity using a troubleshooting container equipped with tools like curl. Let’s use the image brainupgrade/hello:net for both HR and Payroll applications and brainupgrade/tshoot for connectivity testing.

Use the following commands to deploy the application in respective namespaces.

kubectl run hr-app --image=brainupgrade/hello:net --port=8080 --namespace=hr

kubectl run payroll-app --image=brainupgrade/hello:net --port=8080 --namespace=payroll

Expose applications internally using following commands. This allows the applications to be reachable within the cluster.

kubectl expose pod hr-app --port=8080 --target-port=8080 --namespace=hr

kubectl expose pod payroll-app --port=8080 --target-port=8080 --namespace=payroll

Testing Network Policy: Connectivity Check

Deploy a troubleshooting pod in each namespace and attempt to curl the other service.

kubectl run hr-tshoot --image=brainupgrade/tshoot --namespace=hr

Attempt to Connect to Payroll App from HR Troubleshooting Pod:

kubectl exec hr-tshoot --namespace=hr -- curl -s payroll-app.payroll.svc.cluster.local:8080

This should fail based on the network policy setup, as HR should not initiate communication with Payroll.

Deploy Troubleshooting Pod in payroll Namespace:

kubectl run payroll-tshoot --image=brainupgrade/tshoot --namespace=payroll

Attempt to Connect to HR App from Payroll Troubleshooting Pod:

kubectl exec payroll-tshoot --namespace=payroll -- curl -s hr-app.hr.svc.cluster.local:8080

This should succeed, as the network policy allows Payroll to initiate communication with HR. After testing, remove the test applications and troubleshooting pods:

kubectl delete pod,service --all --namespace=hr

kubectl delete pod,service --all --namespace=payroll

These steps provide a practical approach to test the network policies, ensuring that the connectivity aligns with the defined rules, thereby verifying the isolation and access control as intended.

Securing Control Plane

In Kubernetes, Transport Layer Security (TLS) is a cornerstone of the security architecture, ensuring safe and reliable communications within the cluster. TLS provides three key security benefits, including:

Encryption: TLS encrypts data in transit, protecting sensitive information exchanged between Kubernetes components, such as the API server, kubelets, and controllers, from interception and eavesdropping. This encryption is crucial, especially when data moves over untrusted networks.

Authentication: Through TLS certificates, Kubernetes components authenticate each other’s identities, establishing trust within the cluster. For instance, kubelets authenticate the API server and vice versa, ensuring that they are communicating with legitimate components. This mutual authentication prevents unauthorized access and spoofing attacks.

Data Integrity: TLS guarantees the integrity of data being transferred within the cluster. It ensures that the data sent is exactly what is received, without any unauthorized modifications, thus maintaining the reliability of cluster operations and communications.

The Kubernetes control plane leverages TLS extensively:

The API server uses TLS for all interactions, securing communications with etcd, kubelets, and user clients.

Communications between the API server and etcd are encrypted with TLS, safeguarding all cluster state and configuration data.

Node-to-node communications and component-to-component interactions are also TLS-protected, reinforcing the overall security of the cluster.

Implementing TLS, Kubernetes fortifies its control plane and node communications, which is fundamental to maintaining a secure container orchestration environment. This approach aligns with best practices in network security, providing confidentiality, authentication, and integrity for all critical communications within the cluster.

Audit Logging

Kubernetes implements audit logging to keep a record of the sequence of actions or events that affect the cluster. This is essential for compliance, monitoring, and troubleshooting. Here’s how it works:

Audit Policy Configuration:

Kubernetes audit logging behavior is determined by an audit policy. This policy defines what events should be logged and at what level of detail.

The audit policy is set up via a file, often named audit-policy.yaml, which specifies rules for logging different kinds of requests (read, write, and more).

API Server Configuration:

The audit policy is then passed to the API server upon startup.

The API server flags --audit-policy-file and --audit-log-path are used to define the path of the audit policy file and the log file, respectively.

Additionally, parameters like --audit-log-maxage, --audit-log-maxbackup, and --audit-log-maxsize can be configured to manage the log file’s lifecycle.

Event Recording:

When a request is made to the API server, it’s compared against the audit policy to determine if it should be logged.

If the event matches a rule in the audit policy, details such as the user who made the request, the timestamp, the action taken, and the outcome are recorded in the audit log.

Log Backend:

Kubernetes supports different backends for audit logs: logs can be written to a file, sent over a network to an external service, or both.

The most common approach is to write logs to a file, which can then be ingested into a log management system for analysis and monitoring.

Log Analysis:

Audit logs can be analyzed manually or with tools to detect anomalies, investigate incidents, or ensure compliance with policies.

Tools such as Elasticsearch, Logstash, and Kibana (ELK Stack), or Splunk are often used to aggregate and analyze audit logs.

Through audit logging, Kubernetes provides a detailed and customizable logging mechanism, crucial for security and operational oversight. Proper configuration and management of audit logs are key to leveraging their full potential for security and compliance purposes.

Example - Audit Policy

A simple example of an audit-policy.yaml file for Kubernetes would define the rules for what kinds of events should be logged and at what level of detail. Here’s a basic example (Figure 15.3):

Figure 15.3: Simple Audit Policy

This policy file consists of three rules, including:

Log Metadata for Read and Write Requests: The first rule logs metadata (such as the user, timestamp, resource, and verb) for all read and write operations across all namespaces.

Exclude Logging for kube-system Namespace: The second rule prevents logging for any operations in the kube-system namespace.

Log Request and Response Bodies for Pods in Default Namespace: The third rule logs complete request and response bodies (detailed information) for operations on pods in the default namespace.

This is a simple example to illustrate the concept. In a production environment, your audit policy might need to be more complex and tailored to your specific logging and security requirements.

Conclusion

Kubernetes’ security is multi-faceted, encompassing Pod Security Policies, Role-Based Access Control, and Network Policies. The shift towards Pod Security Standards introduces a more streamlined approach, offering enforce, audit, and warn modes for flexible policy implementation. Effective use of these tools, combined with TLS for secure communication and robust audit logging, fortifies the security posture of Kubernetes environments, ensuring both the protection of cluster resources and compliance with security best practices.

In the next chapter, we will delve into the essential tactics for swiftly addressing issues in Kubernetes, particularly vital in production environments. We will explore various Kubernetes API methods for inspecting objects and container logs, accessing object events, and launching debugging containers. Some key topics include the use of commands such as Describe, Debug, Exec, along with insights into Endpoints and Events. This chapter is designed to equip you with the necessary tools and knowledge to quickly and effectively resolve issues in your Kubernetes deployments.

Multiple Choice Questions

What is the primary role of the Kubernetes API Server in cluster security?

Data storage

Network traffic control

Central control plane component

Resource allocation

Which method is NOT a standard Kubernetes API Server authentication mechanism?

Client certificates

Bearer tokens

OAuth

Basic authentication

What is the purpose of Kubernetes Network Policies?

Resource limitation

Traffic control and pod isolation

Data encryption

Access control

Which of these is not a mode available in Pod Security Standards?

Enforce

Audit

Warn

Monitor

What does RBAC stand for in Kubernetes?

Runtime-Based Access Control

Role-Based Access Control

Resource-Based Access Control

Read-Write Access Control

What do Admission Controllers in Kubernetes do?

Control pod scheduling

Monitor network traffic

Intercept and modify API requests

Manage cluster storage

What is the primary function of audit logs in Kubernetes?

Error reporting

Performance monitoring

Record of events for security and analysis

User activity tracking

Which of these is not a predefined Pod Security Standard profile?

Privileged

Baseline

Restricted

Secure

Answers

c

c

b

d

b

c

c

d

Questions

How does the centralized role of the Kubernetes API Server impact cluster security?

What are the implications of incorrectly configured RBAC in a Kubernetes environment?

How do Network Policies contribute to the overall security and integrity of a Kubernetes cluster?

What challenges might arise from the implementation and analysis of audit logs in Kubernetes?

In what scenarios would you choose each of the Pod Security Standard profiles (Privileged, Baseline, Restricted)?

Key Terms

API Server: The central management entity in a Kubernetes cluster that processes API requests and maintains cluster state.

RBAC (Role-Based Access Control): A method for managing access to Kubernetes resources based on user or group roles.

Network Policies: Rules that govern the ingress and egress traffic between pods in a Kubernetes cluster.

Audit Logs: Records of events in Kubernetes, documenting what happened, by whom, and when, for security and troubleshooting.

Pod Security: Practices and policies to secure pods, including controlling their access and capabilities in the cluster.

Authentication: The process of verifying the identity of a user or system in a Kubernetes cluster.

Authorization: The process of determining if a specific authenticated user or group has permissions to perform an action in Kubernetes.

Admission Controllers: Plugins that intercept, validate, or mutate requests to the Kubernetes API server before the request is processed.

CHAPTER 16

Troubleshooting

Introduction

The previous chapter extensively covered the critical aspects of safeguarding a Kubernetes environment. It delved into the roles of the API server in central control and security, the implementation and importance of RBAC for access control, and the significance of Network Policies in traffic management and pod isolation. The chapter also emphasized the role of audit logs in maintaining accountability and the transition to Pod Security Standards for enhancing pod security.

In this chapter, we will cover the essential techniques for diagnosing and resolving issues in Kubernetes clusters, including the health checks of cluster components and performance management. We will delve into troubleshooting application failures, focusing on pod errors, service connectivity, and effective use of logs and monitoring. The chapter also addresses debugging network issues, encompassing network policies, DNS resolution, ingress, and load balancing challenges. Practical approaches such as proficient use of `kubectl` commands, understanding Kubernetes events, and configuration verification will be emphasized to equip you with comprehensive troubleshooting skills in Kubernetes environments.

Structure

In this chapter, we will cover the following topics:

Overview

Troubleshooting Scenarios

Overview

Troubleshooting skills are crucial in managing Kubernetes and cloud computing environments due to their inherent complexity and the critical need for system reliability. Efficient troubleshooting ensures the smooth operation of these systems and enables quick resolution of issues, which is essential for maintaining high availability. This is particularly important in a business context, where system uptime directly impacts customer satisfaction and trust. Moreover, the ability to diagnose and resolve problems rapidly minimizes downtime, a key factor in fast-paced, technology-driven industries. Downtime can have significant financial and reputational repercussions, making effective troubleshooting a vital skill for Kubernetes and in general IT professionals.

Troubleshooting Scenarios

Preparing for the CKA exam through troubleshooting scenarios is advantageous as it emphasizes real-world skills, deepens understanding of Kubernetes components, offers hands-on experience in a live cluster, covers a diverse range of topics, and enhances problem-solving skills. This approach helps you build confidence, practice time management, and aligns well with the exam’s performance-based format.

Now, throughout this chapter, we will go through many scenarios, expected tasks for those and solution details. Solutions section contain the commands you may execute in order to resolve the issue mentioned in the scenarios. You may choose to launch a KIND based Kubernetes cluster to experiment with the following scenarios. In real life, and in the certification exam too, you may be expected to work on many clusters and through the following scenarios you will encounter it.

Scenario 1: Network Policy Verification

Scenario:

In Cluster2, you have a service frontend-service in the namespace dev that should be accessible by pods in the same namespace. However, reports suggest that pods in the dev namespace are unable to access frontend-service. Verify if a network policy is blocking the access and, if so, identify which policy.

Expected Tasks:

Switch to Cluster2.

List all network policies in the dev namespace.

Identify any network policy that might be blocking access to frontend-service.

Solution

kubectl config use-context Cluster2

Correctly switches your context to Cluster2.

kubectl get networkpolicies -n dev

This command lists all network policies in the dev namespace, allowing you to see if there are any policies that might affect frontend-service.

kubectl describe networkpolicy -n dev <network-policy-name>

Once you identify a suspicious network policy from the list, this command will give you the details of that policy. Replace <network-policy-name> with the actual name of the policy you want to inspect.

kubectl get service frontend-service -n dev -o yaml

This is a good additional step. It provides the YAML output of the frontend-service, which can be useful for verifying if the service and its selectors are correctly configured.

Based on these commands, you should be able to determine whether a network policy is causing access issues to frontend-service in the dev namespace of Cluster2.

Scenario 2: Resource Quota Management

Scenario:

In Cluster3, the development team reported that they are unable to deploy any new pods in the staging namespace. You suspect this might be due to a resource quota limitation. Your task is to check the resource quotas in the staging namespace and determine if they are hindering the deployment of new pods.

Expected Tasks:

Switch to Cluster3.

Check the resource quotas set in the staging namespace.

Determine if the current resource usage is near or at the quota limits, which might be preventing new pod deployments.

Solution:

kubectl config use-context Cluster3

Correctly switches your context to Cluster3.

kubectl get resourcequotas -n staging

This command lists the resource quotas set in the staging namespace, which will help you identify if there are any quotas that might affect the deployment of new pods.

kubectl describe resourcequota -n staging <resourcequota-name>

After identifying a specific resource quota, this command provides detailed information about it. Replace <resourcequota-name> with the name of the quota you wish to inspect. This step will reveal whether the current resource usage is close to or at the quota limits, potentially preventing new pod deployments.

These commands should enable you to determine if a resource quota in the staging namespace of Cluster3 is the reason behind the inability to deploy new pods.

Scenario 3: Investigating Node Issues

Scenario:

You’ve been alerted to an issue in Cluster5 where certain nodes are not scheduling any pods. Your task is to identify the nodes that are having issues and determine the cause.

Expected Tasks:

Switch to Cluster5.

List all nodes and their statuses.

Investigate the nodes that are not scheduling pods to understand the reason.

Solution:

kubectl config use-context Cluster5

This switches your context to Cluster5, as required.

kubectl get nodes

This command lists all nodes in the cluster along with their statuses. It helps in quickly identifying any nodes that might be having issues.

kubectl describe node <node-name>

By describing a specific node (replacing <node-name> with the actual node name), you can gather detailed information about its condition, events, and other relevant metrics. This step is crucial for diagnosing why a node might not be scheduling pods.

With these commands, you should be able to identify and investigate any nodes in Cluster5 that are experiencing issues with scheduling pods.

Scenario 4: Service and Ingress Troubleshooting

Scenario:

In Cluster1, there is a web application deployed with a Service of type LoadBalancer named web-service in the prod namespace. Recently, users have reported that they cannot access the application via its external URL. You suspect an issue with either the Service or the Ingress configuration.

Expected Tasks:

Switch to Cluster1.

Check the status of the web-service service in the prod namespace.

Examine the Ingress resources in the prod namespace to verify their configuration.

Inspect the logs of the Ingress controller for potential issues affecting traffic routing.

Consider any potential network policies that might be affecting the connectivity and ensure the pods linked to the web-service are running as expected.

Solution:

kubectl config use-context Cluster1

This switches your context to Cluster1.

kubectl get service -n prod web-service

This command checks the status of the web-service in the prod namespace, helping identify any issues with the service itself.

kubectl get ingresses -n prod

This lists all the Ingress resources in the prod namespace, allowing you to check their configurations and ensure they are set up correctly.

kubectl logs -n ingress-controller <ingress-controller-pod-name>

By inspecting the logs of the Ingress controller, you can identify any routing issues or errors. Replace <ingress-controller-pod-name> with the actual name of the pod running the Ingress controller.

kubectl get networkpolicies -n prod

This command will list all network policies in the prod namespace, which is crucial to understand if any policies are impacting connectivity to the web-service.

kubectl get pods -n prod -l app=<your-app-label>

kubectl describe pods -n prod <pod-name>

These commands help verify if the pods linked to the web-service are running as expected. Replace <your-app-label> and <pod-name> with the actual label and pod name.

Scenario 5: Persistent Volume Troubleshooting

Scenario:

In Cluster2, a Stateful application db-app in the data namespace is experiencing issues with data persistence. The application uses a PersistentVolumeClaim (PVC) named db-pvc. You suspect that the PVC is not properly bound to the required PersistentVolume (PV) or there might be an issue with the underlying storage.

Expected Tasks:

Switch to Cluster2.

Check the status of the PVC db-pvc in the data namespace.

Investigate the corresponding PV and ensure it is properly bound and healthy.

Check the node to which the PV is attached for any storage-related issues.

Verify the logs of the db-app pods to see if there are any errors related to data access.

If the PV is bound to a node, use SSH to access that node and inspect the disk space and mount points to ensure the storage is functioning correctly.

Solution

kubectl config use-context Cluster2

Correctly sets your context to Cluster2.

kubectl get pvc -n data db-pvc

This checks the status of the db-pvc in the data namespace, a crucial first step in diagnosing PVC-related issues.

kubectl get pv <pv-name>

Once you know the name of the PersistentVolume bound to the PVC, this command will give you the status and details of the PV. Replace <pv-name> with the actual name of the PV.

kubectl describe pv <pv-name>

Provides detailed information about the specified PV, including its health and binding status.

kubectl describe node <node-name>

Helps in inspecting the node to which the PV is attached for any storage-related issues. The node name can usually be found in the output of the kubectl describe pv command.

kubectl logs -n data <db-app-pod-name>

By checking the logs of the db-app pods, you can look for any errors or warnings related to data access. Replace <db-app-pod-name> with the actual name of the pod.

ssh <node-name>

SSH into the node that the PV is attached to.

df -h : Check the disk space usage to ensure there’s enough space and the filesystem is not full.

lsblk : List block devices, which helps in confirming the storage devices and their mount points.

Scenario 6: ConfigMap and Secret Management

Scenario:

In Cluster3, an application config-app in the config-test namespace is not behaving as expected. It’s supposed to read configuration data from a ConfigMap and sensitive information from a Secret. You suspect there might be an issue with how these configurations are being passed to the application.

Expected Tasks:

Switch to Cluster3.

Verify the contents of the ConfigMap used by config-app in the config-test namespace.

Check the contents of the Secret associated with config-app.

Ensure that the config-app Deployment is correctly referencing the ConfigMap and Secret.

Check for any recent updates to the ConfigMap or Secret that might have affected the application.

Investigate if the application’s pods are being restarted frequently, which might indicate issues in accessing or applying configuration data.

Solution:

kubectl config use-context Cluster3

Correctly sets your context to Cluster3.

kubectl get configmap -n config-test <configmap-name> -o yaml

Retrieves the contents of the specified ConfigMap in YAML format. Replace <configmap-name> with the actual name of the ConfigMap.

kubectl get secret -n config-test <secret-name> -o yaml

Retrieves the contents of the Secret. Note: Secrets are base64 encoded, so you won’t see plain text values here. Replace <secret-name> with the actual name of the Secret.

kubectl get deployment -n config-test config-app -o yaml

Checks if the config-app Deployment is correctly referencing the ConfigMap and Secret. This is key to ensuring the application is configured correctly.

kubectl get configmap,secret -n config-test -o custom-columns=NAME:.metadata.name,UPDATED:.metadata.creationTimestamp

Lists ConfigMaps and Secrets with their update timestamps, helping identify any recent changes that might have affected the application.

kubectl get pods -n config-test -l app=config-app

Lists the pods for config-app to check for frequent restarts.

kubectl describe pods -n config-test <config-app-pod-name>

Provides detailed information about the pod, including restart count. Replace <config-app-pod-name> with the actual pod name.

kubectl logs -n config-test <config-app-pod-name>

Retrieves logs from the pod that may indicate configuration-related issues.

This comprehensive approach should help you diagnose and resolve any issues with ConfigMap and Secret management for the config-app in Cluster3.

Scenario 7: Role-Based Access Control (RBAC) Configuration

Scenario:

In Cluster4, a team member is having trouble accessing resources in the hr namespace. They should have read-only access to all resources in this namespace. Your task is to ensure the Role-Based Access Control (RBAC) configuration is correctly set up for this user.

Expected Tasks:

Switch to Cluster4.

Verify the existing roles and role bindings in the hr namespace.

Check if the user has the appropriate role binding for read-only access.

If necessary, create or modify the role and role binding to grant the user correct access.

Ensure the roles and role bindings do not grant more permissions than necessary, adhering to the principle of least privilege.

Solutions:

kubectl config use-context Cluster4

Correctly sets your context to Cluster4.

kubectl get roles,rolebindings -n hr

This command lists all roles and role bindings in the hr namespace, allowing you to review existing permissions.

kubectl get rolebinding -n hr <user-role-binding-name>

Checks the specific role binding for the user. Replace <user-role-binding-name> with the actual name of the role binding associated with the user.

If necessary, create or modify the role and role binding to grant the user correct access.

Example: Create a read-only role

kubectl apply -f - <<EOF

apiVersion: rbac.authorization.k8s.io/v1

kind: Role

metadata:

namespace: hr

name: read-only-role

rules:

- apiGroups: [""]

resources: ["pods", "services", "configmaps", "secrets", "deployments", "pvc", "ingresses"]

verbs: ["get", "list", "watch"]

EOF

Example: Create or update a role binding for the user

kubectl apply -f - <<EOF

apiVersion: rbac.authorization.k8s.io/v1

kind: RoleBinding

metadata:

namespace: hr

name: user-read-only-binding

subjects:

- kind: User

name: <user-name>

apiGroup: rbac.authorization.k8s.io

roleRef:

kind: Role

name: read-only-role

apiGroup: rbac.authorization.k8s.io

EOF

The YAML definitions for creating a new read-only role and role binding are correctly structured. They grant read-only access to essential resources like pods, services, configmaps, and so on, adhering to the principle of least privilege.

Scenario 8: Service and Network Policy Configuration

Scenario:

In Cluster5, you have an application api-server running in the dev namespace. This application should be accessible only from pods within the same namespace. However, you’ve received reports that the api-server is accessible from other namespaces as well. Your task is to restrict the access to this service using Network Policies.

Expected Tasks:

Switch to Cluster5.

Verify the current Network Policies applied in the dev namespace.

Create or update a Network Policy to restrict access to the api-server service so that only pods within the dev namespace can access it.

Validate the configuration to ensure it adheres to the required access restrictions.

Check if the api-server service is correctly configured with the right selectors and ports to align with the network policy.

Solution:

kubectl config use-context Cluster5

Sets the context to Cluster5.

kubectl get networkpolicies -n dev

Lists the existing Network Policies in the dev namespace, a necessary step to understand the current network access configurations.

kubectl apply -f - <<EOF

apiVersion: networking.k8s.io/v1

kind: NetworkPolicy

metadata:

namespace: dev

name: restrict-api-access

spec:

podSelector:

matchLabels:

app: api-server

ingress:

- from:

- podSelector: {}

EOF

The YAML definition for creating or updating the Network Policy (restrict-api-access) is well-constructed. It specifies that only pods within the dev namespace (denoted by podSelector: {} under ingress) can access the api-server pods.

kubectl describe networkpolicy -n dev restrict-api-access

Describes the newly created or updated Network Policy to validate its configuration.

kubectl get service -n dev api-server -o yaml

Retrieves the configuration of the api-server service in YAML format. This step is crucial to ensure that the service selectors and ports are aligned with the Network Policy.

This sequence of commands should effectively restrict access to the api-server service in the dev namespace of Cluster5 to only the pods within the same namespace, adhering to the specified network security requirements.

Scenario 9: Securing Sensitive Data with Secrets

Scenario:

In Cluster6, there’s an application secure-app in the finance namespace that requires access to sensitive data like API keys and passwords. However, you’ve noticed that these sensitive data are currently stored as plain text in a ConfigMap. Your task is to securely manage this sensitive data using Kubernetes Secrets.

Expected Tasks:

Switch to Cluster6.

Locate and inspect the ConfigMap currently holding sensitive data.

Convert the sensitive data into a Secret and ensure its encoded correctly.

Update the secure-app deployment to use the Secret instead of the ConfigMap.

Validate that the secure-app pods are using the Secret and functioning correctly.

Ensure that the Secret is not exposed in logs or command outputs, maintaining the confidentiality of the sensitive data.

Solution

kubectl config use-context Cluster6

Sets the context to Cluster6.

kubectl get configmap -n finance <configmap-name> -o yaml

Retrieves the existing ConfigMap that is currently holding the sensitive data. Replace <configmap-name> with the actual name of the ConfigMap.

kubectl create secret generic -n finance sensitive-secret \

--from-literal=api-key=<api-key-value> \

--from-literal=password=<password-value>

Ensure to replace <api-key-value> and <password-value> with the actual sensitive values. Remember, the command as written exposes these values in the command line, which might be logged. In a real-world scenario, consider a more secure method of creating secrets.

kubectl patch deployment -n finance secure-app \

--patch "$(cat <<EOF

spec:

template:

spec:

volumes:

- name: sensitive-data

secret:

secretName: sensitive-secret

EOF

)"

Updates the secure-app deployment to use the Secret instead of the ConfigMap. The patch command correctly modifies the deployment to mount the Secret.

kubectl get pods -n finance -l app=secure-app

kubectl describe pods -n finance <secure-app-pod-name>

These commands verify that the secure-app pods are up and running and using the new configuration. Replace <secure-app-pod-name> with the actual name of one of the pods.

kubectl get deployment -n finance secure-app -o yaml | grep sensitive-secret

This command checks the deployment configuration to ensure the Secret is being used without exposing its content.

Scenario 10: Kubernetes Cluster Upgrade

Scenario:

You are tasked with upgrading the Kubernetes version of Cluster1. The current version is v1.20, and you need to upgrade it to v1.21. It’s critical to plan and execute the upgrade with minimal disruption to the running services.

Expected Tasks:

Switch to Cluster1.

Check the current version of the cluster.

Plan and execute the upgrade of the control plane nodes.

Upgrade the worker nodes in a rolling fashion to minimize downtime.

Verify the upgrade by checking the version post-upgrade and ensure all nodes are Ready.

Ensure to drain and cordon the nodes appropriately during the upgrade process to prevent service disruption.

Solution:

kubectl config use-context Cluster1

Correctly sets the context to Cluster1.

kubectl version

This checks the current version of Kubernetes running in the cluster, an important initial step before planning an upgrade.

kubeadm upgrade plan

followed by

kubeadm upgrade apply v1.21.x

These commands plan and apply the upgrade to the control plane nodes. Replace v1.21.x with the specific minor version you’re upgrading to.

For upgrading worker nodes:

kubectl drain <worker-node-name> --ignore-daemonsets

Safely drains the worker node, moving the running pods to other nodes. Replace <worker-node-name> with the actual name of each worker node.

After performing the necessary upgrade steps on the node (such as updating kubelet), you uncordon the node using kubectl uncordon <worker-node-name>. This step makes the node schedulable again.

kubectl get nodes && kubectl version

These commands verify the version of Kubernetes post-upgrade and check the status of all nodes to ensure they are ready and healthy.

Scenario 11: Setting Up and Troubleshooting Cluster Monitoring

Scenario:

In Cluster2, you need to set up monitoring to track the performance and health of the cluster. However, after setting up a monitoring solution (let’s say Prometheus), you notice that it’s not collecting data as expected. Your task is to ensure proper monitoring setup and resolve any issues with data collection.

Expected Tasks:

Switch to Cluster2.

Verify the installation and status of the Prometheus deployment.

Check if Prometheus is correctly configured to scrape metrics from the cluster nodes and pods.

Investigate and resolve any issues preventing Prometheus from collecting data.

Validate the setup by ensuring Prometheus is now correctly gathering and displaying cluster metrics.

Ensure that the monitoring setup adheres to security best practices, such as restricting access to monitoring dashboards and encrypting data in transit.

Solution:

kubectl config use-context Cluster2

Correctly sets the context to Cluster2.

kubectl get deployment -n monitoring prometheus-server

Checks the status of the Prometheus deployment in the monitoring namespace, a crucial initial step for verifying the monitoring setup.

Checking Prometheus configuration:

kubectl get servicemonitors --all-namespaces

Lists ServiceMonitors across all namespaces:

kubectl describe servicemonitor <servicemonitor-name> -n <namespace>

Provides detailed information about a specific ServiceMonitor, which is essential for verifying if Prometheus is correctly configured to scrape metrics.

kubectl logs -n monitoring <prometheus-pod-name>

Inspecting the Prometheus pod logs is vital for troubleshooting any issues related to data collection. Replace <prometheus-pod-name> with the actual name of the Prometheus pod.

Validating the setup:

Access the Prometheus UI and, if used, Grafana, to check metrics and dashboards. This step ensures that Prometheus is correctly gathering and displaying cluster metrics.

Scenario 12: Optimizing Application Resource Usage

Scenario:

In Cluster3, an application analytics-app deployed in the data-analysis namespace is experiencing performance issues. Initial analysis suggests that the application might not be efficiently utilizing cluster resources. Your task is to optimize the resource usage of this application to improve its performance.

Expected Tasks:

Switch to Cluster3.

Analyze the current resource requests and limits set for analytics-app.

Adjust the resource requests and limits based on the application’s usage patterns and performance needs.

Monitor the application’s performance after the adjustments to ensure improvements.

Ensure that these changes do not negatively impact other applications in the cluster.

Consider implementing Horizontal Pod Autoscaling (HPA) for analytics-app to automatically adjust the number of pods based on CPU or memory usage.

Solution:

kubectl config use-context Cluster3

Sets the context to Cluster3.

kubectl describe deployment -n data-analysis analytics-app | grep "Limits" -A 2

This command reviews the current resource requests and limits set for analytics-app, an essential step in understanding its resource utilization.

kubectl edit deployment -n data-analysis analytics-app

Editing the deployment allows you to adjust the resource requests and limits based on the application’s performance needs and usage patterns.

Monitoring Performance Post-Adjustment:

kubectl top pod -n data-analysis

Monitors the resource usage of pods in real-time, which helps in evaluating the effectiveness of the adjustments. Continuously monitoring other applications in the cluster is important to ensure that the changes do not negatively affect their performance.

Additional Complexity - HPA Implementation:

kubectl autoscale deployment -n data-analysis analytics-app --cpu-percent=80 --min=1 --max=10

This command sets up Horizontal Pod Autoscaling for the analytics-app, automatically adjusting the number of pods based on CPU usage. It scales between 1 and 10 pods, triggering scaling when CPU usage reaches 80%.

This sequence of commands and actions should help optimize the analytics-app’s resource usage in Cluster3, enhancing its performance while ensuring overall cluster efficiency and stability.

Scenario 13: Implementing and Testing Network Policies

Scenario:

In Cluster4, you have a frontend application frontend-app and a backend service backend-service, both running in the app namespace. You need to ensure that the frontend-app can communicate with backend-service but block all other incoming and outgoing traffic to backend-service from outside the app namespace.

Expected Tasks:

Switch to Cluster4.

Create a Network Policy to allow traffic from frontend-app to backend-service and restrict all other traffic.

Apply and verify the Network Policy to ensure it is functioning as intended.

Test the connectivity from frontend-app to backend-service and from a pod outside the app namespace to validate the policy.

After implementing the Network Policy, monitor the network traffic to ensure there are no unintended blocks or allowances.

Solution:

kubectl config use-context Cluster4

Correctly sets your context to Cluster4.

kubectl apply -f - <<EOF

apiVersion: networking.k8s.io/v1

kind: NetworkPolicy

metadata:

name: allow-frontend-to-backend

namespace: app

spec:

podSelector:

matchLabels:

app: frontend-app

egress:

- to:

- podSelector:

matchLabels:

app: backend-service

policyTypes:

- Ingress

- Egress

---

apiVersion: networking.k8s.io/v1

kind: NetworkPolicy

metadata:

name: allow-backend-service

namespace: app

spec:

podSelector:

matchLabels:

app: backend-service

ingress:

- from:

- podSelector:

matchLabels:

app: frontend-app

policyTypes:

- Ingress

- Egress

EOF

Above YAML ensures that only frontend-app can communicate with backend-service within the app namespace while restricting other traffic.

kubectl describe networkpolicy -n app allow-frontend-to-backend

This command verifies the details of the applied Network Policy to ensure it matches the intended configuration.

Testing Connectivity:

kubectl exec -n app -it <frontend-app-pod-name> -- curl -s <backend-service pod IP>

Tests connectivity from the frontend-app pod to the backend-service. Replace <frontend-app-pod-name> with the actual pod name.

For testing connectivity from outside the app namespace, the kubectl run command to create a temporary pod in a different namespace is appropriate. Using this pod to attempt a curl to backend-service serves as a negative test to confirm the policy’s effectiveness.

kubectl get networkpolicies -n app && kubectl get events -n app --sort-by=.metadata.creationTimestamp

These commands help monitor network policies and related events to ensure there are no unintended blocks or allowances in network traffic.

Scenario 14: Cluster Backup and Restore

Scenario:

In Cluster5, you are responsible for implementing a disaster recovery plan. This includes creating a backup strategy for the cluster’s critical data, such as the etcd database and Kubernetes resource configurations. Additionally, you need to demonstrate the ability to restore from a backup in case of a disaster.

Expected Tasks:

Switch to Cluster5.

Create a backup of the etcd database.

Backup Kubernetes resource configurations.

Simulate a recovery scenario by restoring from the etcd backup.

Verify that the cluster state is consistent with the state before the simulated disaster.

Ensure that the backup and restore processes are secure and do not expose sensitive data.

Solution:

Switch to Cluster5

kubectl config use-context Cluster5

Create a backup of the etcd database

Example: Take a snapshot of the etcd database

kubectl exec -n kube-system <etcd-pod-name> -- /bin/sh -c "ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /var/lib/etcd/snapshot.db"

kubectl cp -n kube-system <etcd-pod-name>:/var/lib/etcd/snapshot.db /path/to/backup/snapshot.db

Backup Kubernetes resource configurations. Example: Export Kubernetes resources to YAML files:

kubectl get all --all-namespaces -o yaml > cluster-resources-backup.yaml

Simulate a recovery scenario by restoring from the etcd backup. Example: Replace the etcd data directory with the backup:

kubectl cp -n kube-system /path/to/backup/snapshot.db <etcd-pod-name>:/var/lib/etcd/snapshot.db

Login into control plane node and update data-dir flag value (yaml file /etc/kubernetes/manifests/etcd.yaml) to /var/lib/etcd/snapshot.db

After a while, verify that the cluster state is consistent with the state before the simulated disaster.

kubectl get all --all-namespaces

kubectl get nodes

Ensure backup and restore processes are secure. Secure access to etcd snapshots and Kubernetes resource backup files.

chmod 600 /path/to/backup/snapshot.db

chmod 600 cluster-resources-backup.yaml

Scenario 15: Auto-Scaling Based on Custom Metrics

Scenario:

In Cluster6, you have a web application web-app deployed in the frontend namespace. Due to varying traffic, you need to implement an auto-scaling strategy based on custom metrics, specifically the number of HTTP requests per second. Assume that the necessary metrics are already being collected and exposed by Prometheus.

Expected Tasks:

Switch to Cluster6.

Implement Horizontal Pod Autoscaling (HPA) for web-app based on the custom metric (HTTP requests per second).

Apply the HPA and monitor its effect on the scaling of web-app.

Adjust the scaling parameters if necessary to ensure optimal performance during traffic spikes.

Ensure the scaling operation does not cause service disruptions and maintains a balance between performance and resource utilization.

Solution:

Switch to Cluster6

kubectl config use-context Cluster6

Implement Horizontal Pod Autoscaling (HPA) for web-app based on the custom metric

Example: Create an HPA for web-app based on HTTP requests per second:

kubectl create -f - <<EOF

apiVersion: autoscaling/v2

kind: HorizontalPodAutoscaler

metadata:

name: web-app-hpa

namespace: frontend

spec:

scaleTargetRef:

apiVersion: apps/v1

kind: Deployment

name: web-app

minReplicas: 2

maxReplicas: 10

metrics:

- type: Pods

pods:

metric:

name: http_requests

target:

type: AverageValue

averageValue: 1000m

EOF

Apply the HPA and monitor its effect on the scaling of web-app:

kubectl get hpa -n frontend

kubectl describe hpa -n frontend web-app

Adjust the scaling parameters if necessary to ensure optimal performance during traffic spikes:

kubectl edit hpa -n frontend web-app

Scenario 16: Securing Sensitive Data in Transit and at Rest

Scenario:

In Cluster1, you have several applications that handle sensitive data. You need to ensure that this data is secure both in transit within the cluster and at rest. This includes configuring TLS for in-cluster communications and encrypting sensitive data stored in Secrets.

Expected Tasks:

Switch to Cluster1.

Implement TLS for in-cluster communication between services.

Encrypt data stored in Kubernetes Secrets.

Validate that the data in transit is encrypted and that Secrets are stored securely.

Ensure that these security measures do not disrupt normal cluster operations.

Configure network policies to restrict unauthorized access to services handling sensitive data.

Solution:

Switch to Cluster1

kubectl config use-context Cluster1

Implement TLS for in-cluster communication between services. Example: Create or update a Service with a TLS certificate:

kubectl create secret tls my-tls-secret -n <namespace> --cert=<path/to/cert.crt> --key=<path/to/cert.key>

kubectl apply -f - <<EOF

apiVersion: v1

kind: Service

metadata:

name: my-service

namespace: <namespace>

spec:

ports:

- port: 443

targetPort: 8080

selector:

app: my-app

tls:

- secretName: my-tls-secret

EOF

Validate that the data in transit is encrypted and Secrets are stored securely. Access the service and verify the TLS connection:

curl https://my-service.<namespace>.svc

Ensure that these security measures do not disrupt normal cluster operations. Monitor logs, events, and application functionality:

kubectl get events --sort-by=.metadata.creationTimestamp

Configure network policies to restrict unauthorized access:

kubectl apply -f - <<EOF

apiVersion: networking.k8s.io/v1

kind: NetworkPolicy

metadata:

name: restrict-sensitive-services

namespace: <namespace>

spec:

podSelector:

matchLabels:

app: sensitive-app

ingress:

- from:

- podSelector:

matchLabels:

role: trusted

EOF

Scenario 17: Implementing Advanced Scheduling Techniques

Scenario:

In Cluster2, you need to ensure that certain critical applications are scheduled on high-performance nodes, while less critical applications are scheduled on standard nodes. The high-performance nodes are labeled with node-type=high-perf.

Expected Tasks:

Switch to Cluster2.

Create or update Pod specifications for critical applications to ensure they are scheduled on high-performance nodes.

Create or update Pod specifications for less critical applications to prefer standard nodes but allow scheduling on high-performance nodes, if necessary.

Validate that the Pods are being scheduled on the intended nodes.

Monitor the cluster to ensure that the scheduling preferences do not lead to resource contention or imbalance.

Set up pod anti-affinity rules to prevent critical applications from being co-located on the same high-performance node for fault tolerance.

Solution:

Switch to Cluster2

kubectl config use-context Cluster2

Create or update Pod specifications for critical applications to ensure they are scheduled on high-performance nodes. Example: Add nodeSelector to the Pod spec:

kubectl apply -f - <<EOF

apiVersion: v1

kind: Pod

metadata:

name: critical-app

spec:

containers:

- name: app-container

image: critical-image

nodeSelector:

node-type: high-perf

EOF

Create or update Pod specifications for less critical applications to prefer standard nodes. Example: Add nodeAffinity to the Pod spec:

kubectl apply -f - <<EOF

apiVersion: v1

kind: Pod

metadata:

name: less-critical-app

spec:

containers:

- name: app-container

image: less-critical-image

affinity:

nodeAffinity:

preferredDuringSchedulingIgnoredDuringExecution:

nodeSelectorTerms:

- matchExpressions:

- key: node-type

operator: In

values:

- standard

EOF

Validate that the Pods are being scheduled on the intended nodes:

kubectl get pods -o wide

Monitor the cluster to ensure that the scheduling preferences do not lead to resource contention or imbalance. Monitor nodes, resource usage. Note that, metrics-server component needs to be installed in order to get responses to kubectl top commands. Please refer to Chapter 17: Kubernetes Production Essentials for more details.

kubectl top nodes

kubectl top pods

Set up pod anti-affinity rules to prevent critical applications from being co-located on the same high-performance node for fault tolerance. Example: Add pod anti-affinity to the Pod spec:

kubectl apply -f - <<EOF

apiVersion: v1

kind: Pod

metadata:

name: critical-app

spec:

containers:

- name: app-container

image: critical-image

affinity:

podAntiAffinity:

requiredDuringSchedulingIgnoredDuringExecution:

- labelSelector:

matchExpressions:

- key: node-type

operator: In

values:

- high-perf

topologyKey: kubernetes.io/hostname

EOF

Scenario 18: Setting Up Cluster-Wide Logging

Scenario:

In Cluster3, you need to set up a comprehensive logging solution that captures logs from all pods across all namespaces. This setup should enable centralized log management and analysis.

Expected Tasks:

Switch to Cluster3.

Implement a logging solution like Fluentd or a similar tool to aggregate logs from all pods.

Configure the logging solution to forward logs to a centralized log management system (like Elasticsearch).

Validate that logs from all pods are being captured and can be queried in the centralized system.

Ensure that the logging setup does not significantly impact the performance of the cluster.

Set up appropriate resource limits for the logging agents to prevent them from consuming excessive resources.

Solution:

Switch to Cluster3

kubectl config use-context Cluster3

Implement a logging solution (Fluentd in this example). Example: Deploy Fluentd as a DaemonSet to collect logs from all pods:

kubectl apply -f - <<EOF

apiVersion: apps/v1

kind: DaemonSet

metadata:

name: fluentd

namespace: kube-system

spec:

selector:

matchLabels:

app: fluentd

template:

metadata:

labels:

app: fluentd

spec:

containers:

- name: fluentd

image: fluent/fluentd

resources:

limits:

memory: "200Mi"

requests:

memory: "100Mi"

EOF

Configure the logging solution to forward logs to a centralized log management system (like Elasticsearch). Example: If using Elasticsearch, you may need to configure Fluentd to output logs to an Elasticsearch endpoint.

Validate that logs from all pods are being captured and can be queried in the centralized system. Access the centralized log management system (Elasticsearch, Kibana, and more) to query logs

Ensure that the logging setup does not significantly impact the performance of the cluster. Monitor resource usage and performing.

kubectl top nodes

kubectl top pods -n kube-system

Note: Setting up Elasticsearch, Kibana or Fluentd may not be expected during certification tests however setting up of observability setup is key in Kubernetes Cluster management. Please refer to the next chapter for complete setup.

Scenario 20: Configuring Ingress for Service Routing and SSL Termination

Scenario:

In Cluster4, you have multiple web services running in different namespaces. You need to set up an Ingress controller to manage external access to these services, including routing based on URL paths and enabling SSL termination for secure HTTPS connections.

Expected Tasks:

Switch to Cluster4.

Deploy an Ingress controller like Nginx or Traefik if not already present.

Configure Ingress resources to route traffic based on URL paths to the appropriate services.

Set up SSL termination on the Ingress controller for secure HTTPS connections.

Validate that the routing works correctly and that SSL termination is functioning.

Ensure the Ingress configuration adheres to best practices for security and performance.

Solution:

Switch to Cluster4

kubectl config use-context Cluster4

Deploy an Ingress controller (Nginx example). Example: Deploy Nginx Ingress Controller.

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/cloud/deploy.yaml

Configure Ingress resources to route traffic based on URL paths. Example: Create an Ingress resource for a web service.

kubectl apply -f - <<EOF

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

name: web-ingress

namespace: web-namespace

annotations:

nginx.ingress.kubernetes.io/rewrite-target: /

spec:

rules:

- host: example.com

http:

paths:

- path: /service1

pathType: Prefix

backend:

service:

name: service1

port:

number: 80

- path: /service2

pathType: Prefix

backend:

service:

name: service2

port:

number: 80

EOF

Set up SSL termination on the Ingress controller. Example: Create a Secret with an SSL certificate.

kubectl create secret tls web-ingress-tls -n web-namespace --cert=<path/to/cert.crt> --key=<path/to/cert.key>

Update the Ingress resource to enable SSL termination.

kubectl apply -f - <<EOF

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

name: web-ingress

namespace: web-namespace

annotations:

nginx.ingress.kubernetes.io/rewrite-target: /

nginx.ingress.kubernetes.io/ssl-redirect: "true"

spec:

rules:

- host: example.com

http:

paths:

- path: /service1

pathType: Prefix

backend:

service:

name: service1

port:

number: 80

tls:

- hosts:

- example.com

secretName: web-ingress-tls

EOF

Validate that the routing works correctly and SSL termination is functioning. Access the services using the specified paths and check for HTTPS.

Scenario 21: Debugging Cluster Network Issues

Scenario:

In Cluster6, applications in the dev namespace are experiencing intermittent connectivity issues. Your task is to diagnose and resolve any network problems affecting the cluster’s functionality.

Expected Tasks:

Switch to Cluster6.

Inspect the network configurations and policies in the dev namespace.

Check the status and logs of network components such as CNI plugins, kube-proxy, and any network policies that might be affecting connectivity.

Identify and resolve any misconfigurations or issues found in the network setup.

Test the network connectivity to ensure that the issues have been resolved.

Monitor overall network performance to ensure that the resolution doesn’t adversely impact other components or namespaces in the cluster.

Solution:

Switch to Cluster6

kubectl config use-context Cluster6

Inspect the network configurations and policies in the dev namespace:

kubectl get networkpolicies -n dev

kubectl get pods -n dev

kubectl describe networkpolicies -n dev <network-policy-name>

Check the status and logs of network components. Example: Check the status of CNI plugins.

kubectl get pods -n kube-system -l k8s-app=calico-node

kubectl logs -n kube-system calico-node-<pod-id>

Example: Check the status of kube-proxy

kubectl get pods -n kube-system -l k8s-app=kube-proxy

kubectl logs -n kube-system kube-proxy-<pod-id>

Identify and resolve misconfigurations or issues in the network setup. Review logs and error messages to identify potential problems. Adjust network policies, update CNI plugins, or troubleshoot kube-proxy.

Test network connectivity to ensure issues are resolved. Example: Create a temporary pod for testing connectivity:

kubectl run -n dev -it --rm --image=busybox test-pod -- sh

Inside the pod, try to ping or connect to other services.

Scenario 22: Configuring Persistent Storage for Stateful Applications

Scenario:

In Cluster1, you have a stateful application db-app that requires persistent storage. You need to configure a Persistent Volume (PV) and a Persistent Volume Claim (PVC) to ensure data persistence for this application.

Expected Tasks:

Switch to Cluster1.

Create a Persistent Volume that matches the storage requirements of db-app.

Create a Persistent Volume Claim in the namespace where db-app is running.

Update the db-app deployment to use the PVC for its data storage.

Validate that the db-app is correctly using the persistent storage and that data is retained across pod restarts.

Ensure the storage solution is scalable and can be easily expanded to accommodate future data growth.

Solution:

Switch to Cluster1

kubectl config use-context Cluster1

Create a Persistent Volume (PV) that matches the storage requirements of db-app

Example: Define a PV with appropriate capacity, access modes, and storage class.

kubectl apply -f - <<EOF

apiVersion: v1

kind: PersistentVolume

metadata:

name: db-app-pv

spec:

capacity:

storage: 5Gi

accessModes:

- ReadWriteOnce

storageClassName: standard

hostPath:

path: "/data/db-app"

EOF

Create a Persistent Volume Claim (PVC) in the namespace where db-app is running

Example: Define a PVC requesting storage from the created PV.

kubectl apply -f - <<EOF

apiVersion: v1

kind: PersistentVolumeClaim

metadata:

name: db-app-pvc

namespace: db-namespace

spec:

accessModes:

- ReadWriteOnce

storageClassName: standard

resources:

requests:

storage: 5Gi

EOF

Update the db-app deployment to use the PVC for its data storage

Example: Update the volumes and volumeMounts in the deployment spec.

kubectl edit deployment db-app -n db-namespace

Add the following to the volumes section:

- name: db-app-storage

persistentVolumeClaim:

claimName: db-app-pvc

Update the volumeMounts section to mount the volume at the desired path.

Validate that the db-app is correctly using the persistent storage.

Example: Check the pod logs for any initialization or storage-related messages.

kubectl logs -n db-namespace <db-app-pod-name>

Ensure the storage solution is scalable. Consider using dynamic provisioning and a storage class that supports scalability. Update storage class and PVC specifications accordingly.

Scenario 23: Managing Resource Quotas and Namespace Configurations

Scenario:

In Cluster4, you need to manage resources effectively across different namespaces to ensure fair usage and prevent resource hogging by any single namespace. Your task is to configure resource quotas for namespaces and ensure proper resource allocation.

Expected Tasks:

Switch to Cluster4.

Define and apply a Resource Quota to a specific namespace (for example, dev-namespace) to limit the use of CPU and memory resources.

Configure a LimitRange in the namespace to set default CPU and memory requests and limits for pods.

Verify that the Resource Quota and LimitRange are enforced when new pods are deployed in the namespace.

Monitor resource usage in the namespace to ensure compliance with the defined quotas.

Set up alerts or notifications when a namespace reaches a certain percentage of its resource quota.

Solution:

Switch to Cluster4

kubectl config use-context Cluster4

Define and apply a Resource Quota to a specific namespace (for example, dev-namespace)

kubectl apply -f - <<EOF

apiVersion: v1

kind: ResourceQuota

metadata:

name: dev-resource-quota

namespace: dev-namespace

spec:

hard:

pods: "10"

requests.cpu: "1"

requests.memory: 1Gi

limits.cpu: "2"

limits.memory: 2Gi

EOF

Configure a LimitRange in the namespace to set default CPU and memory requests and limits for pods:

kubectl apply -f - <<EOF

apiVersion: v1

kind: LimitRange

metadata:

name: dev-limit-range

namespace: dev-namespace

spec:

limits:

- type: Pod

max:

cpu: "1"

memory: 1Gi

min:

cpu: "100m"

memory: 100Mi

- type: Container

max:

cpu: "1"

memory: 1Gi

min:

cpu: "100m"

memory: 100Mi

EOF

Verify that the Resource Quota and LimitRange are enforced when new pods are deployed in the namespace. Deploy a pod in the dev-namespace and observe any resource allocation violations. Monitor resource usage in the namespace to ensure compliance with the defined quotas. Use kubectl top or other monitoring tools to check resource usage in the dev-namespace.

Scenario 24: Implementing a Blue-Green Deployment Strategy

Scenario:

In Cluster6, you are required to implement a blue-green deployment strategy for a critical application critical-app to ensure zero downtime during updates and easy rollback in case of issues.

Expected Tasks:

Switch to Cluster6.

Set up two versions of critical-app: blue (current version) and green (new version), running simultaneously.

Configure a service to initially route traffic to the blue version.

Gradually shift traffic to the green version while monitoring for issues.

In case of no issues, route all traffic to the green version; otherwise, roll back to the blue version.

Ensure that both blue and green versions are exposed to the same database or state, without causing data inconsistencies or conflicts.

Solution:

Switch to Cluster6

kubectl config use-context Cluster6

Define Blue and Green Deployments

kubectl apply -f - <<EOF

apiVersion: apps/v1

kind: Deployment

metadata:

name: critical-app-blue

spec:

replicas: 3

selector:

matchLabels:

app: critical-app

version: blue

template:

metadata:

labels:

app: critical-app

version: blue

spec:

containers:

- name: critical-app

image: your-registry/critical-app:blue

---

apiVersion: apps/v1

kind: Deployment

metadata:

name: critical-app-green

spec:

replicas: 3

selector:

matchLabels:

app: critical-app

version: green

template:

metadata:

labels:

app: critical-app

version: green

spec:

containers:

- name: critical-app

image: your-registry/critical-app:green

EOF

Define a Service to initially route traffic to the blue version:

kubectl apply -f - <<EOF

apiVersion: v1

kind: Service

metadata:

name: critical-app-service

spec:

selector:

app: critical-app

version: blue

ports:

- protocol: TCP

port: 80

targetPort: 8080

EOF

Gradually shift traffic to the green version using labels and selector updates. Monitor the application for issues during the transition. Route all traffic to the ‘Green’ version if there are no issues.

kubectl apply -f - <<EOF

apiVersion: v1

kind: Service

metadata:

name: critical-app-service

spec:

selector:

app: critical-app

version: green

ports:

- protocol: TCP

port: 80

targetPort: 8080

EOF

In case of issues, roll back to the blue version:

kubectl apply -f - <<EOF

apiVersion: v1

kind: Service

metadata:

name: critical-app-service

spec:

selector:

app: critical-app

version: blue

ports:

- protocol: TCP

port: 80

targetPort: 8080

EOF

Conclusion

In this chapter, we delved into a variety of practical scenarios to enhance skills in diagnosing and resolving issues within Kubernetes clusters. This chapter covered essential aspects such as network troubleshooting, resource quota management, pod security, persistent storage configurations, and advanced deployment strategies. Through these scenarios, we learned how to enforce network policies, manage sensitive data, implement effective deployment strategies like blue-green deployments, and handle complex networking and storage issues. The chapter emphasized the importance of balancing security, performance, and resource efficiency, providing insights into best practices for maintaining robust and reliable Kubernetes environments. The hands-on approach adopted in this chapter is crucial for developing the intuition and technical prowess necessary for adept Kubernetes administration, ensuring preparedness for real-world challenges and the Certified Kubernetes Administrator (CKA) exam. Please note that the scenarios covered in this chapter don’t resemble the format and questions employed by CNCF CKA exam authority.

In the next chapter, we will elevate our Kubernetes cluster to a production-grade level by integrating essential third-party plugins. We’ll explore the installation and configuration of critical tools such as cert-manager for HTTPS management, Nginx Kubernetes Ingress Controller for efficient traffic routing, Metrics Server for Horizontal Pod Autoscaling, the EFK stack (Elasticsearch, Fluentd, Kibana) for log analytics, and Prometheus and Grafana for comprehensive metrics monitoring. This chapter will be instrumental in enhancing the cluster’s capabilities, security, and performance monitoring, essential for real-world Kubernetes deployments.

Points to Remember

Effective Diagnostic Techniques: Mastering kubectl commands and understanding Kubernetes events are crucial for diagnosing issues in clusters, applications, and network configurations.

Resource Management: Skillfully managing resource quotas and understanding pod resource allocations are essential to prevent and resolve issues related to resource constraints and contention.

Network Troubleshooting: Proficiency in debugging network issues, including understanding network policies, DNS resolution within Kubernetes, and connectivity problems, is vital for maintaining cluster health.

Storage and Stateful Applications: Effective management of persistent storage, including configuring Persistent Volumes (PVs) and Persistent Volume Claims (PVCs), is key to ensuring data persistence and handling stateful applications in Kubernetes.

Application Deployment Strategies: Understanding and implementing advanced deployment strategies such as blue-green deployments ensures zero downtime during application updates and facilitates easy rollback in case of issues.

Multiple Choice Questions

Which command is used to view the events of a specific Kubernetes namespace?

kubectl get events -n <namespace>

kubectl describe events -n <namespace>

kubectl logs -n <namespace>

What does the CrashLoopBackOff status of a pod typically indicate?

The pod is in the process of restarting.

The pod repeatedly crashes after restarting.

The pod is currently being scaled down.

Which resource type can be used to restrict the creation of privileged pods?

NetworkPolicy

PodSecurityPolicy

LimitRange

If a pod is unable to access an external URL, what could be a potential cause?

Incorrect Persistent Volume configuration

Network Policy restricting egress traffic

Insufficient CPU allocation

What is the purpose of a LimitRange in Kubernetes?

To restrict the total resource consumption of a namespace

To set default resource requests and limits for pods in a namespace

To define the maximum number of pods that can run in a namespace

Which of the following is a primary tool for troubleshooting pod networking issues?

kubectl describe

kubectl exec

kubectl logs

How can you check the resource requests and limits of a specific pod?

kubectl top pod <pod-name>

kubectl describe pod <pod-name>

kubectl get pod <pod-name> -o yaml

What is a common reason for pods to be stuck in the ‘Pending’ state?

Insufficient resources on the nodes

Network policies blocking communication

Incompatible image version

Which Kubernetes object is crucial for routing external traffic to services inside the cluster?

DaemonSet

Ingress

StatefulSet

To debug an application’s runtime issues, which of the following should be examined?

Node metrics

Pod logs

PersistentVolumeClaims

Answers

a

b

b

b

b

a

b

a

b

b

Questions

How would you approach a situation where a Kubernetes service is accessible from within the cluster but not from external sources? What factors would you consider in your troubleshooting process?

Consider a scenario where a set of pods in a Kubernetes deployment are frequently restarting. What could be some potential causes, and how would you systematically diagnose and address the issue?

What strategies would you employ to troubleshoot a persistent volume that is failing to bind to a persistent volume claim? How would you ensure data integrity and minimal downtime in such situations?

If you encounter a situation where certain pods are consistently evicted or fail to schedule, what Kubernetes resource limitations or configurations would you investigate first?

How would you differentiate and diagnose networking issues caused by CNI misconfigurations versus those caused by incorrect network policies? What tools or commands would be most useful in each case?

Key Terms

Pod Lifecycle: The states and transitions in a pod’s lifecycle, including Pending, Running, Succeeded, Failed, and Unknown.

Resource Quotas: Resource Quotas to limit resource consumption (CPU, memory, storage) within a namespace.

Network Policies: Network Policies for controlling the flow of traffic to and from pods and services.

Persistent Volumes (PV) and Persistent Volume Claims (PVC): PVs and PVCs for managing persistent storage.

Horizontal Pod Autoscaler (HPA): HPA for automatically scaling pods in a deployment based on metrics like CPU utilization.

Ingress Controllers: Ingress Controllers for managing external access to services in a cluster, typically HTTP.

Pod Security Admission Controller: Pod Security Admission Controller helps define and enforce security standards for pods.

DaemonSets: DaemonSets for ensuring that all or some nodes run a copy of a pod, typically for cluster-wide services like logging and monitoring.

CHAPTER 17

Kubernetes Production Essentials

Introduction

In the previous chapter, we delved into advanced Kubernetes troubleshooting, focusing on diagnosing and resolving issues related to pod lifecycle, network policies, persistent storage, and resource management. We explored effective strategies for managing pod security in light of the deprecation of PodSecurityPolicy, the implementation of resource quotas, and the intricacies of network and service connectivity. The chapter also covered the utilization of Horizontal Pod Autoscalers and DaemonSets for efficient cluster operation, emphasizing the importance of continuous monitoring and adaptive management in maintaining a robust Kubernetes environment.

In this chapter, we will focus on transforming a basic Kubernetes cluster into a production like environment by integrating key third-party tools. This includes the NGINX Ingress Controller for traffic management, Cert-Manager for automated TLS certificate handling, and the Metrics Server for resource monitoring. We will also implement the EFK Stack (Elasticsearch, Fluent-bit, Kibana) for advanced logging, integrate Application Performance Monitoring tools for real-time application insights, and use Prometheus and Grafana for comprehensive monitoring and analytics. These enhancements, alongside additional tools for scalability, security, and resilience, are critical in elevating the cluster’s capabilities for production readiness.

Structure

In this chapter, we will cover the following topics:

Overview

Production Essentials - Extensions

CNI - Calico

Metrics Server

Ingress Controller

Certificate Manager

Prometheus and Grafana

Elasticsearch, Fluent-bit and Kibana, Elastic APM

Overview

Elevating a vanilla Kubernetes cluster to production-grade involves integrating key tools for enhanced performance, security, and observability. Metrics Server provides resource usage insights for autoscaling and performance tuning. Ingress Controllers manage external traffic routing, while Cert-Manager automates TLS certificate management, bolstering security. Prometheus and Grafana offer comprehensive monitoring and advanced data visualization for operational intelligence. Lastly, the EFK stack (Elasticsearch, Fluentd, Kibana) along with Elastic APM (Application Performance Monitoring) enables efficient log aggregation and analysis, crucial for distributed tracing and debugging in complex environments. Together, these tools form a robust ecosystem for maintaining a reliable, secure, and highly observable Kubernetes infrastructure.

Production Essentials - Extensions

In this chapter, we’ll guide you through setting up essential tools such as CNI - Calico, Metrics Server, Ingress Controller, Cert-Manager, Prometheus and Grafana, and EFK. However, consider additional tooling tailored to your specific business needs and context. Your cluster’s architecture should align with your operational goals and requirements, ensuring optimal performance and security. Some of the tools like EFK used in this chapter may require more involved setup in the production and covering that is beyond the scope of this book.

CNI Plugin

Choosing the right Container Network Interface (CNI) plugin in Kubernetes is critical for ensuring optimal network performance, scalability, and security. Different CNIs offer varying features and are suited to different scales and types of workloads. The choice impacts integration with existing infrastructure, particularly in cloud environments. A well-supported CNI with a strong community ensures reliability and ease of management. This decision is integral to aligning Kubernetes deployments with strategic business and technical objectives.

Calico is a popular open-source networking and network security solution for containers, virtual machines, and native host-based workloads. It provides high-performance, scalable Layer 3 networking with no overlay for Kubernetes and other orchestrators. Calico supports a rich set of network security features including network policies for fine-grained control, enhancing the security of containerized environments. It’s known for its simplicity and ease of use, offering seamless integration with various cloud providers and Kubernetes environments. Additionally, Calico can be used in hybrid and multi-cloud deployments, making it versatile for diverse infrastructural needs.

In Chapter 14: Kubernetes CNI, we covered the installation of Calico.

Metrics Server

The Metrics Server in Kubernetes is a scalable, efficient source of container resource metrics. These metrics, such as CPU and memory usage, provide insights for auto scaling applications and for the Horizontal Pod Autoscaler (HPA). It collects resource metrics from Kubelets and exposes them via the Kubernetes API server for use by tools such as kubectl. Unlike Heapster, its predecessor, Metrics Server is lighter and more performance-focused, storing only the latest values to reduce storage demands. It’s an essential component for real-time monitoring and performance analysis in Kubernetes clusters. Metrics Server is particularly valuable in dynamic environments where resource allocation needs constant adjustment.

Installation

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

Minor tweak: For local cluster like KIND, we need to add the flag to the program argument: --kubelet-insecure-tls

Use following command and add the flag to the main program.

kubectl edit deploy -n kube-system metrics-server

Verification

Execute the following commands to validate its working. If you get metrics info, then all is ok.

kubectl top nodes

kubectl top pods

Ingress Controller

An Ingress Controller in Kubernetes is a daemon, deployed as a pod, that manages external access to services within a cluster. It handles HTTP/HTTPS routing, directing traffic to the appropriate backend services based on defined Ingress resources. This is essential for providing externally-reachable URLs, load balancing, SSL termination, and name-based virtual hosting. It acts as a reverse proxy and a load balancer, becoming a central point through which all inbound traffic flows. Ingress Controllers are extendable and customizable, with popular options including NGINX, HAProxy, and Traefik. Their deployment simplifies the process of exposing services and managing traffic rules in a Kubernetes environment.

Installation

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/cloud/deploy.yaml

Verification

Run the following command to see all the pods are running.

kubectl get pods -n ingress-nginx

Certificate Manager

Cert-manager in Kubernetes is an automation tool for managing and issuing TLS certificates. It simplifies the process of obtaining, renewing, and using SSL/TLS certificates for Kubernetes applications. Cert-manager supports multiple sources for certificate issuance, including Let’s Encrypt, HashiCorp Vault, and self-signed certificates. It provides a Kubernetes-native way to specify and manage certificates, integrating seamlessly into the Kubernetes API. Through Custom Resource Definitions (CRDs), it allows users to define and manage certificates declaratively. Cert-manager thus ensures secure communication within Kubernetes clusters and for services exposed to the Internet, enhancing overall security.

Installation

helm repo add jetstack https://charts.jetstack.io

helm repo update

helm install cert-manager jetstack/cert-manager --namespace cert-manager --set installCRDs=true --create-namespace

Configuration

Cert-manager requires definition of ClusterIssuer resource to validate domain name. For clusters which are hosted in cloud and publicly accessible, you can use the ClusterIssuer using the following commands:

wget https://raw.githubusercontent.com/brainupgrade-in/book-kubernetesadmin/main/chapter_17/01-cluster-issuer-http.yaml

Replace <YOUR-EMAIL> with your valid email id:

sed -i ‘s/your-email@example.com/<YOUR-EMAIL>/g’ 01-cluster-issuer-http.yaml

kubectl apply -f 01-cluster-issuer-http.yaml

Since our cluster is KIND based and in private network behind firewall, we will use DNS resolver in ClusterIssuer resource using the following commands:

wget https://raw.githubusercontent.com/brainupgrade-in/book-kubernetesadmin/main/chapter_17/02-cluster-issuer-dns.yaml

Replace <YOUR-EMAIL> with your valid email id:

sed -i 's/your-email@example.com/<YOUR-EMAIL>/g' 02-cluster-issuer-dns.yaml

Replace <YOUR_SECRET_ACCESS_KEY_ID> and <YOUR_SECRET_ACCESS_KEY> with your Route 53 Access Key and ID in the following commands:

sed -i 's/YOUR_ACCESS_KEY_ID/<YOUR_SECRET_ACCESS_KEY_ID>/g' 02-cluster-issuer-dns.yaml

kubectl create secret generic route53-secret-access-key --from-literal=secret-access-key=<YOUR_SECRET_ACCESS_KEY> --namespace cert-manager

kubectl apply -f 02-cluster-issuer-dns.yaml

Verification

Create ingress using the following command:

kubectl create ingress app -n default --rule="k8s.gheware.com/?(.*)=app:80,tls=secre8shh" --class nginx --annotation=cert-manager.io/cluster-issuer=letsencrypt-prod-dns --annotation=nginx.ingress.kubernetes.io/rewrite-target=/\$1

And check if the certificate shows READY=true using the following command:

kubectl get certificate

Prometheus and Grafana

Prometheus is an open-source monitoring and alerting toolkit widely used in cloud and microservices environments. It’s known for its efficient time series database and powerful query language (PromQL), enabling detailed and accurate insights into systems and applications. Prometheus works by scraping metrics from configured targets at specified intervals, evaluating rule expressions, displaying results, and triggering alerts if conditions are met. It’s highly scalable and integrates well with Kubernetes, making it a preferred choice for monitoring containerized applications. The ecosystem of Prometheus includes support for multiple modes of graphing and dashboarding, notably Grafana. It’s simple yet powerful architecture makes it a key component in many DevOps toolchains for continuous monitoring and observability.

Grafana is a popular open-source analytics and interactive visualization web application. It provides rich ways to visualize time series data and metrics through beautiful and dynamic dashboards. Grafana is commonly used in conjunction with monitoring tools such as Prometheus and InfluxDB, allowing users to query, visualize, alert on, and understand their metrics. It supports a wide range of data sources, each with its own specific query editor. Grafana’s user-friendly interface and extensive customization options make it a favorite for creating detailed and insightful graphical representations of data. Its capabilities extend beyond just monitoring, including support for analytics and IoT applications, making it a versatile tool for data visualization needs.

Installation

Add repo of prometheus and grafana using the following commands:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

helm repo update

Install prometheus in the monitoring namespace:

kubectl create namespace monitoring

helm install prometheus prometheus-comemunity/prometheus --namespace monitoring

To pre-configure grafana with default datasource as prometheus and few useful and publicly available dashboards, download the values yaml and input the same while installing grafana. Run grafana service as NodePort for easy access.

wget https://raw.githubusercontent.com/brainupgrade-in/dockerk8s/main/misc/observability/values.yaml

helm install grafana grafana/grafana --set service.type=NodePort --namespace monitoring --values values.yaml

Obtain grafana URL using the following command:

echo "http://$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' k8s-prod-control-plane):$(kubectl get svc -l app.kubernetes.io/name=grafana -ojsonpath='{.items[0].spec.ports[0].nodePort}')"

The default username is admin and password is obtained using the following command:

kubectl get secret --namespace monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

Verification

Access the grafana URL and check the preloaded dashboards.

EFK and Elastic APM

EFK, comprising Elasticsearch, Fluent-bit, and Kibana, is a powerful logging and monitoring solution tailored for Kubernetes environments. Elasticsearch efficiently stores and indexes log data, Fluent-bit aggregates logs from various sources, and Kibana provides a user-friendly interface for data visualization. This stack is crucial in Kubernetes for its ability to handle large volumes of distributed log data, offering insights into application performance and aiding in troubleshooting. EFK’s comprehensive logging and analysis capabilities are essential for maintaining the health and reliability of containerized applications and infrastructure.

Elastic Application Performance Monitoring (APM) is a tool within the Elastic Stack designed to monitor software applications in real time and analyze their performance. It provides detailed insights into application errors, response times, and throughput, helping developers identify and fix issues quickly. Elastic APM supports various programming languages and frameworks, and integrates deeply with Elasticsearch for advanced data analysis and visualization. This integration allows teams to correlate application performance metrics with other data in their Elastic Stack, enabling comprehensive observability across their systems for distributed tracing.

Installation

Using the following commands, you can install Elasticsearch, Fluent-bit and Kibana.

helm install es oci://registry-1.docker.io/bitnamicharts/elasticsearch --set master.persistence.size=1Gi --set data.persistence.size=1Gi --set master.replicaCount=1 --set data.replicaCount=1 --namespace elasticsearch --create-namespace

helm install kibana oci://registry-1.docker.io/bitnamicharts/kibana --set elasticsearch.hosts[0]=es-elasticsearch.elasticsearch.svc --set elasticsearch.port=9200 --set service.type=NodePort --namespace elasticsearch --create-namespace

kubectl apply -f https://raw.githubusercontent.com/brainupgrade-in/book-kubernetesadmin/main/chapter_17/03-fluent-bit.yaml

To install Elastic APM, run the following command:

kubectl apply -f https://raw.githubusercontent.com/brainupgrade-in/dockerk8s/main/app/weather/07a-apm-server.yaml

Verification

Forward the kibana port to localhost on port 5601 (http://localhost:5601) using the following command and if you see the kibana dashboard on your browser, you have EFK setup done.

kubectl port-forward svc/kibana 5601:5601

Conclusion

In this chapter, we explored the integration and utilization of key monitoring and logging tools within a Kubernetes environment to elevate a cluster to production-grade readiness. We covered the setup and configuration of the Metrics Server, Ingress Controller, Cert-Manager, Prometheus and Grafana, and the EFK stack (Elasticsearch, Fluentd, Kibana), highlighting their roles in enhancing cluster performance, security, and observability. The chapter emphasized the importance of robust monitoring for application performance and system health, and detailed how these tools collectively provide a comprehensive view of the cluster’s operational status. Practical examples and commands illustrated the deployment and troubleshooting processes.

In the next chapter, we will concentrate on establishing observability for a microservices-based application within a Kubernetes cluster. Using a weather application composed of three microservices as a case study, the chapter will demonstrate the practical steps involved in configuring observability. This example will serve as a guide to understand the intricacies and best practices in monitoring and managing microservices in a Kubernetes environment. The chapter aims to provide a comprehensive approach to achieving effective observability, essential for maintaining and optimizing microservice architectures.

Multiple Choice Questions

What is the primary function of the Metrics Server in Kubernetes?

Deploying applications

Collecting resource metrics for autoscaling

Managing user authentication

Storing persistent data

Which tool is used for managing external access to services in Kubernetes?

Ingress Controller

Cert-Manager

Prometheus

Cert-Manager in Kubernetes is used for:

Monitoring container performance

Managing TLS certificates

Load balancing

What does the EFK stack stand for?

Elasticsearch, Fluent Bit, Kibana

ElasticSearch, Fluent Bit, Kubernetes

ElasticSearch, Filebeat, Kibana

Prometheus in Kubernetes is primarily used for:

Database management

Monitoring and alerting

Traffic routing

Certificate management

Which tool is typically used with Prometheus for data visualization?

Fluentd

Grafana

Ingress Controller

Cert-Manager

In the context of Kubernetes, what is an Ingress Controller responsible for?

Node management

HTTP/HTTPS routing

Data encryption

Which of the following is a benefit of using Cert-Manager?

Auto-scaling pods

Automating certificate renewal

Load balancing

Fluentd in the EFK stack is primarily used for:

Data visualization

Log aggregation

Certificate management

Which of these is not a part of the EFK stack?

Elasticsearch

Kibana

File Bit

Answers

b

a

b

a

b

b

b

b

b

c

Questions

How does the integration of tools like Metrics Server, Ingress Controller, Cert-Manager, Prometheus and Grafana, and the EFK stack in a Kubernetes environment contribute to the overall system reliability and efficiency?

Considering the diverse functionalities of these tools, what are the potential challenges in configuring and maintaining them in a production-grade Kubernetes cluster, and how can these challenges be mitigated?

How does the implementation of Cert-Manager within a Kubernetes cluster enhance its security posture, and what are the implications of not automating TLS certificate management in such environments?

In what ways can the data visualized through Grafana, when combined with the log insights provided by the EFK stack, be leveraged to make strategic decisions about scaling, resource allocation, and performance optimization in Kubernetes?

Key Terms

Metrics Server: A scalable, efficient source of container resource metrics in Kubernetes, crucial for auto scaling and performance monitoring.

Ingress Controller: A Kubernetes tool that manages external access to cluster services, handling HTTP/HTTPS routing and providing features like SSL termination and name-based virtual hosting.

Cert-Manager: A native Kubernetes certificate management controller, automating the management, issuance, and renewal of TLS certificates for secure communication.

Prometheus: An open-source monitoring system and time series database, widely used in Kubernetes for gathering and processing metrics, alerting, and operational insight.

EFK Stack: An acronym for Elasticsearch, Fluent-bit, and Kibana, representing a powerful logging and monitoring setup in Kubernetes for log aggregation, storage, and visualization.

CHAPTER 18

Microservices Observability

Introduction

In the preceding chapter, we delved into the crucial aspect of enhancing the capabilities of our Kubernetes cluster by leveraging a range of production essential tools. These tools, which played a pivotal role in optimizing our cluster’s performance, included Metrics Server, Ingress Controller, EFK (Elasticsearch, Fluent-bit, Kibana), APM Server, as well as Prometheus and Grafana. Through this exploration, we laid the foundation for a more robust and efficient Kubernetes environment.

This chapter delves into the essential aspects of monitoring and managing Kubernetes environments. It emphasizes the importance of observability in Kubernetes, covering best practices, guidelines, and tools. The majorareas include logging, distributed tracing, and metrics which are crucial for ensuring the health and efficiency of Kubernetes clusters. This chapter serves as a comprehensive guide for Kubernetes practitioners to maintain robust observability in their microservices architecture.

Structure

In this chapter, we will discuss the following topics:

Overview

Observability

Tools Setup

Event Logs - EFK

Distributed Tracing - APM Server

Metrics - Prometheus and Grafana

Overview

Observability is a key aspect of modern software architectures, especially in microservices environments. In this chapter, we will explore setting up observability for a Weather Application consisting of three microservices: Database, API Services, and Web Frontend. We’ll use Elasticsearch, Fluent-bit, Kibana, APM Server, Prometheus, and Grafana for this purpose.

Observability

Observability, a concept originally from control system engineering, is defined as the ability to deduce a system’s internal states from its external outputs. In simpler terms, it’s about having sufficient insights into a system to enable effective remedial actions.

The three fundamental pillars of observability are:

Event Logs: These are records of events that occur within the system. Events are distinct occurrences with metadata that provide context about the system at the time of their occurrence.

Tracing: In a system composed of numerous parts or components that collectively deliver specific functionalities, it’s crucial to trace the flow of requests and responses across these distributed components. This tracing is essential for efficient debugging and problem-solving.

Metrics: These are measurements of system performance over time and are indicative of the system’s service levels.

Tools Setup

Before diving into the observability process, ensure the following tools are set up in your Kubernetes environment:

Elasticsearch: A distributed, RESTful search and analytics engine capable of addressing a growing number of use cases.

Fluent-bit: A lightweight log processor and forwarder, which allows you to collect data/logs from different sources.

Kibana: A free and open user interface that lets you visualize your Elasticsearch data and navigate the Elastic Stack.

APM Server: Integrates with Elasticsearch and Kibana to provide performance monitoring and distributed tracing capabilities.

Prometheus: An open-source systems monitoring and alerting toolkit.

Grafana: An open-source platform for monitoring and observability.

Event Logs - EFK

In Chapter 17: Kubernetes Production Essentials, we had setup these tools, however following is a quick refresher on these tools setup. Run the following commands:

helm install es oci://registry-1.docker.io/bitnamicharts/elasticsearch --set master.persistence.size=1Gi --set data.persistence.size=1Gi --set master.replicaCount=1 --set data.replicaCount=1 --namespace elasticsearch --create-namespace

helm upgrade --install fluent-bit fluent/fluent-bit \

--namespace fluent-bit --create-namespace \

--set backend.type=es \

--set backend.es.host=es-elasticsearch-master-hl.elasticsearch.svc \

--set backend.es.port=9200 \

--set dashboards.enabled=true

Update fluent-bit config (kubectl edit cm fluent-bit -n fluent-bit) to reflect following values (as shown in Figure 18.1)

Host es-elasticsearch-master-hl.elasticsearch.svc

Suppress_Type_Name On

Figure 18.1: Fluent Bit configuration - ES host and Type

helm install kibana oci://registry-1.docker.io/bitnamicharts/kibana --set elasticsearch.hosts[0]=es-elasticsearch.elasticsearch.svc.cluster.local --set elasticsearch.port=9200 --set elasticsearchHosts="es-elasticsearch.elasticsearch.svc:9200" --set service.type=NodePort --namespace elasticsearch --create-namespace

Check the nodeport of kibana service (kubectl get svc kibana -n elasticsearch) and access the kibana dashboard using the following URL. Replace the NODE-PORT with the actual value.

http://172.18.0.2:<NODE-PORT>/app/management/data/index_management/indices

You will see that the logstash index is already populated since fluent-bit is pushing the logs to Elasticsearch using this index (refer to Figure 18.2).

Figure 18.2: Index - logstash

Additionally, you can access the Discover endpoint (http://172.18.0.3:<NODE-PORT>/app/discover#/), to perform log analytics using the available options on this page (refer to Figure 18.3).

Figure 18.3: Kibana Log Analytics Dashboard

Distributed Tracing - APM Server

The Application Performance Monitoring (APM) Server is a central component in the Elastic Stack, designed to ingest performance data from your applications and transform it into structured APM events. It serves as a bridge between application agents and Elasticsearch, enabling real-time monitoring and analysis of application performance. Distributed log tracing, a key feature of APM, allows for the tracking of requests as they travel through various microservices in a distributed system. This tracing provides a comprehensive view of interactions and performance bottlenecks, essential for effective debugging and optimizing microservices architectures. The integration of APM Server in a microservices setup enhances the visibility and understanding of complex system behaviors.

Installation - APM Server

kubectl apply -f https://raw.githubusercontent.com/brainupgrade-in/book-kubernetesadmin/main/chapter_18/01-apm-server.yaml

Now let us deploy weather application into the cluster before we explore APM Dashboard.

Weather Application - Architecture

Weather Application provides weather information for the given city. This application contains three primary microservices, including:

Database Microservice: This component is responsible for storing and managing the weather data. It acts as the backbone of the application, providing necessary data to other services.

API Services Microservice: This is the intermediary layer that processes requests from the Web Frontend. It fetches the relevant data from a third-party weather service and ensures that the necessary information is returned to the frontend.

Web Frontend Microservice: This component serves as the user interface. It allows users to interact with the application, such as querying for weather forecasts. It sends requests to the API Services and displays the received data to the users.

Installation - Weather Application

Setup configuration and secrets for the app using the following commands:

kubectl apply -f https://raw.githubusercontent.com/brainupgrade-in/book-kubernetesadmin/main/chapter_18/02-weather-config-secret.yaml

Using the following commands, deploy all the three microservices for weather application:

kubectl apply -f https://raw.githubusercontent.com/brainupgrade-in/book-kubernetesadmin/main/chapter_18/03-weather-db.yaml

kubectl apply -f https://raw.githubusercontent.com/brainupgrade-in/book-kubernetesadmin/main/chapter_18/04-weather-api-services.yaml

kubectl apply -f https://raw.githubusercontent.com/brainupgrade-in/book-kubernetesadmin/main/chapter_18/05-weather-frontend.yaml

Verify that the APM server and Weather microservices are running fine using the following command:

kubectl get pods -A

Once all are running fine, in a matter of a few minutes, the APM server will start pushing the performing data to the elasticsearch. Head over to APM Dashboard and you will see the Weather microservices on the dashboard (refer to Figure 18.4):

http://172.18.0.3:<NODE-PORT>/app/apm/services

Figure 18.4: APM Dashboard

Now access the application by forwarding its port on the localhost using the following command and you will see the Weather Web UI (refer to Figure 18.5):

kubectl port-forward -n weather svc/weather 8888:80

Figure 18.5: Weather Web UI - frontend

Key in any city, say Bangalore, and you will see an error (refer to Figure 18.6)

Figure 18.6: App error

To trace this, visit the APM - Services - transactions section on the APM Dashboard and at the bottom of the page you will notice the three methods as transactions. Click WeatherController#setWeather transaction and scroll to the bottom of the page to view the traces of request (refer to Figure 18.7).

Figure 18.7: Distributed Tracing

Clicking the last trace (GET www.metaweather.com), you will see the full error details (refer to Figure 18.8).

Figure 18.8: Root cause

Click the "View span in Discover" and you will be taken to the application log analytics screen (refer to Figure 18.9).

Figure 18.9: Log Trace

Metrics - Prometheus and Grafana

Metrics monitoring comprises not only Cluster monitoring but also application monitoring. Prometheus collects metrics data on the cluster control components as well as data provided by the applications. Applications expose their monitoring data in prometheus format using available plugins, spring boot prometheus plugin in this case since weather application microservices are developed using spring boot java.

Installation - Prometheus and Grafana

helm install prometheus prometheus-community/prometheus --namespace monitoring --create-namespace

wget https://raw.githubusercontent.com/brainupgrade-in/book-kubernetesadmin/main/chapter_18/06-grafana-values.yaml

helm install grafana grafana/grafana --set service.type=NodePort --namespace monitoring --values 06-grafana-values.yaml

Obtain password using the following command (username is admin):

kubectl get secret --namespace monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

Obtain grafana node port using the following command:

kubectl get svc -n monitoring grafana

Metrics - Application

Access the grafana dashboard using the following URL. Replace NODE-PORT with the actual value:

http://172.18.0.2:<NODE-PORT>/

Click Create new dashboard (or this link http://172.18.0.2<NODE-PORT>/dashboard/new?utm_source=grafana_gettingstarted&orgId=1) and you will see an option to import a dashboard. Enter 12685, click load and import this dashboard buttons while selecting Prometheus datasource, publicly available dashboard for spring boot apps (refer to Figure 18.10):

Figure 18.10: Weather app - metrics

Metrics - Cluster

Explore the Dashboard folder section and you will find many cluster related dashboards. One such is Node view (URL: http://172.18.0.2:<NODE-PORT>/d/k8s_views_nodes/kubernetes-views-nodes). This dashboard provides a great detail of insights into the cluster operations (refer to Figure 18.11).

Figure 18.11: Metrics monitoring - Cluster

Conclusion

This chapter comprehensively covered the critical aspects of implementing observability in microservices architecture using Kubernetes. We explored the significance of tools such as Elasticsearch, Fluent-bit, Kibana, APM Server, Prometheus, and Grafana in achieving a robust observability strategy. Through practical demonstrations, we highlighted the importance of the three pillars of observability - logs, tracing, and metrics - in understanding and optimizing the performance of a Weather Application. The chapter also delved into the intricacies of APM Server and distributed log tracing, showcasing their pivotal roles in monitoring and debugging complex distributed systems. This knowledge is invaluable for professionals seeking to enhance the reliability, performance, and scalability of their microservices deployments, ultimately leading to more resilient and efficient IT systems.

In the upcoming chapter, our focus will shift to leveraging Kubernetes to create an auto-scalable CI/CD service using Jenkins. We’ll delve into the methods and strategies for harnessing Kubernetes’ capabilities to enhance the scalability and efficiency of Jenkins in continuous integration and continuous deployment processes. This exploration will provide insights into optimizing Jenkins for dynamic, scalable environments.

Points to Remember

Understanding Observability: This chapter emphasized the importance of observability in microservices, particularly in a Kubernetes environment, and its role in maintaining system health and performance.

Three Pillars of Observability: The chapter highlighted the three pillars - Logs, Tracing, and Metrics - as fundamental to achieving comprehensive observability, each providing unique insights into the system’s operations.

Real-Time Data Analysis: It emphasized the importance of real-time data analysis in observability, enabling quick response to issues and efficient system performance optimization.

APM Server and Distributed Tracing: The role of APM Server in application performance monitoring and the significance of distributed log tracing in identifying and resolving performance bottlenecks in a microservices architecture were thoroughly discussed.

Multiple Choice Questions

What are the three main pillars of observability in a microservices environment?

Logs, Tracing, Metrics

Deployment, Scaling, Monitoring

APIs, Databases, Frontend Services

Networking, Storage, Computing

Which tool is used for log processing and forwarding in a Kubernetes environment?

Elasticsearch

Fluent-bit

Grafana

Jenkins

What is the primary role of the APM Server in observability?

Managing database connections

Visualizing log data

Orchestration of microservices

Ingesting and analyzing application performance data

Which tool is primarily used for visualizing metrics dashboards in observability setups?

Kibana

Prometheus

Grafana

Docker

In the context of Kubernetes, what is the benefit of implementing observability?

Increasing the number of microservices

Enhancing system reliability and performance

Simplifying code development

Reducing the cost of cloud storage

Answers

a

b

d

c

b

Questions

How does the implementation of observability tools like Elasticsearch, Fluent-bit, Kibana, APM Server, Prometheus, and Grafana in a Kubernetes environment contribute to the overall efficiency and reliability of microservices architecture?

What are the potential challenges in integrating and configuring the three pillars of observability (Logs, Tracing, Metrics) in a complex system like the Weather Application, and how can these challenges be effectively overcome?

Considering the rapid evolution of technology and increasing complexity of systems, what are the future prospects and potential developments in the field of observability, particularly in relation to Kubernetes and microservices?

Key Terms

Observability: The ability to infer the internal state of a system based on its external outputs, crucial for understanding and monitoring microservices architectures.

Application Performance Monitoring (APM) Server: A tool that ingests performance data from applications, vital for performance analysis and monitoring in observability.

Distributed Tracing: A method for tracking request and response flow through various components in a distributed system, essential for debugging and performance optimization.

Grafana: An open-source platform for monitoring and observability, used for visualizing metrics, logs, and traces, facilitating in-depth analysis of system performance.

CHAPTER 19

Scalable Jenkins on Kubernetes

Introduction

The preceding chapter provided an in-depth exploration of implementing observability in microservices architecture using Kubernetes, emphasizing the use of tools such as Elasticsearch, Fluent-bit, Kibana, APM Server, Prometheus, and Grafana. It demonstrated the critical importance of the three observability pillars - logs, tracing, and metrics - particularly in the context of a Weather Application, and delved into APM Server and distributed log tracing for effective monitoring and debugging. It is a valuable resource for professionals aiming to improve the reliability, performance, and scalability of microservices systems.

This chapter will provide an in-depth guide on establishing a scalable Jenkins setup, emphasizing two main strategies: leveraging external storage and utilizing pods for job execution. It will delve into the use of the Jenkins Kubernetes plugin, which facilitates the creation of pods for building jobs. The chapter methodically demonstrates this process through a detailed step-by-step setup, followed by practical testing using a straightforward inline pipeline.

This chapter uses a KIND based kubernetes cluster. You can delete earlier created clusters, if any, create a new KIND cluster having minimum 2 nodes before running examples referred in this chapter.

Structure

In this chapter, we will discuss the following topics:

Overview

Scalable Jenkins

Overview

Jenkins is an open-source automation server widely used for continuous integration and continuous delivery (CI/CD) in software development. It automates various stages of a delivery pipeline, from building, testing to deploying code. Jenkins supports a vast ecosystem of plugins, allowing it to integrate with virtually any tool in the CI/CD toolchain.

Scalable Jenkins CI/CD on Kubernetes works by dynamically creating Jenkins agent pods within the cluster for each build job, optimizing resource utilization and handling varying workloads efficiently. Kubernetes’ native scalability features, like auto-scaling and load balancing, ensure the system adapts to changing demands. Pod templates in Jenkins define different build environments, allowing for parallel execution of diverse jobs. The Jenkins Kubernetes plugin automates pod management, seamlessly integrating Jenkins with Kubernetes’ orchestration capabilities. This approach maximizes efficiency, reduces infrastructure costs, and enhances the overall agility of the CI/CD pipeline.

Scalable Jenkins

Scalable Jenkins is primarily achieved through two key strategies: the use of external storage and the utilization of pods for building jobs. External storage ensures data persistence and scalability, independent of the Jenkins server. Utilizing pods for job execution leverages Kubernetes’ orchestration capabilities, allowing dynamic resource allocation and efficient load handling. This combination enhances the scalability and resilience of Jenkins in handling CI/CD pipelines, especially in cloud-native environments. Overall, these strategies lead to a more robust, flexible, and scalable Jenkins setup.

Let us now install jenkins, provide permissions to deploy and manage the apps and configure its Cloud plugin to run jobs within pods.

Installation - Jenkins

kubectl create ns jenkins

kubectl apply -f https://github.com/OrangeAVA/Ultimate-Certified-Kubernetes-Administrator-CKA-Certification-Guide/blob/main/chapter_19/01-jenkins.yaml

Obtain the password using the following command. Save the password for later use.

kubectl logs -f -n jenkins jenkins-0

Access the Jenkins web UI by running the following command and the UI is accessible on http://localhost:8888

kubectl port-forward -n jenkins svc/jenkins 8888:8080

Once you enter the password, select the second option (Select plugins to install) and you will be shown the next screen. Click None at the top before clicking Install button.

Figure 19.1: Plugin Installation - None

Click Skip and continue as admin link at the bottom, then Save and Finish button. Once you click Start using Jenkins button, you will be taken to the main Jenkins dashboard. The docker image used in this StatefulSet contains many advanced plugins including Cloud Kubernetes plugin for efficient CICD.

Access Control

Create a service account and cluster role.

kubectl create serviceaccount jenkins --namespace=jenkins

kubectl create clusterrole jenkins-deployer --verb=create,delete,get,list,patch,update,watch --resource=deployments,services,pods,pods/log,pods/exec

Assign this cluster role to jenkins which is running as StatefulSet in the jenkins namespace. Provide access to deploy cluster wide to the jenkins app and assign this service account to the jenkins app.

kubectl create clusterrolebinding jenkins-deployer-binding --clusterrole=jenkins-deployer --serviceaccount=jenkins:jenkins

kubectl set sa sts jenkins jenkins -n jenkins

Wherein,

kubectl set sa sts <name of StatefulSet> <name of service account> -n <namespace name>

Once sa (service account) is set, the jenkins app will be restarted.

Cloud Kubernetes Plugin setup

Next, configure a cloud either by visiting this endpoint http://localhost:8080/cloud/new or clicking Configure a cloud link on the front page (refer to Figure 19.2).

Figure 19.2: Configure new cloud

Now configure the cloud named 'kubernetes' (http://localhost:8888/cloud/kubernetes/configure) and selecting the WebSocket option, enter Jenkins URL and click Save (refer to Figure 19.3).

Figure 19.3: Configure the cloud

Now create a new job (http://localhost:8888/view/all/newJob) named test and type as Pipeline. Click Ok to configure. In the pipeline section, add the following pipeline as code and click to build (refer to Figure 19.4).

podTemplate {

node(POD_LABEL) {

stage('Run shell') {

sh 'echo hello world'

}

}

}

Figure 19.4: Inline pipeline script

In a few moments, the first build of this test job will be Success. Browse, http://localhost:8888/job/test/1/console, for more details and you will notice that a Pod was launched to build this job (refer to Figure 19.5).

Figure 19.5: Pod for build Job

We used a simple pipeline to demonstrate that pods are used to build the jobs, thus providing scalability to your Jenkins.

Jenkins Kubernetes plugin offers sophisticated features for advanced pipelines. You can find more advanced pipelines here - https://github.com/brainupgrade-in/jenkins/tree/main/examples/03-kubernetes

Conclusion

This chapter effectively demonstrated how to set up a scalable Jenkins environment, focusing on the utilization of external storage and pod-based job execution within a Kubernetes framework. Through step-by-step instructions and a practical example involving an inline pipeline, it provides a clear, actionable roadmap for enhancing Jenkins’ performance and flexibility in cloud-native ecosystems.

In the next chapter, we will look at GitOps using ArgoCD and GitHub. GitOps is a DevOps approach that uses Git as a single source of truth and automation tool for managing and deploying software infrastructure and applications. ArgoCD, a Kubernetes-native tool, automatically synchronizes the cluster’s state with configurations stored in GitHub repositories. This setup ensures consistent, traceable, and automated deployments. It enhances collaboration, reliability, and security in the software delivery process.

Points to Remember

Focus on Scalability: This chapter emphasizes the importance of scalability in Jenkins setups, particularly through the use of external storage and Kubernetes pods, ensuring Jenkins can efficiently handle varying loads and workflows.

Utilization of the Jenkins Kubernetes Plugin: The chapter highlights the critical role of the Jenkins Kubernetes plugin in enabling pod-based job execution, facilitating dynamic resource allocation and streamlined job management within a Kubernetes environment.

Integration with Kubernetes: This chapter details the integration of Jenkins with Kubernetes, showcasing how Kubernetes’ orchestration capabilities can be leveraged to enhance the automation, scalability, and reliability of Jenkins CI/CD pipelines.

Multiple Choice Questions

What is the primary purpose of using external storage in a scalable Jenkins setup?

To increase the processing speed of Jenkins

To ensure data persistence and scalability

To enhance the security of Jenkins data

To reduce the cost of storage

What does the Jenkins Kubernetes plugin primarily facilitate?

Integration with cloud services

Dynamic creation of pods for Jenkins jobs

Automated code reviews

Encryption of sensitive data

In a scalable Jenkins setup, what role do pod templates play?

They define the UI layout of Jenkins

They dictate the version of Jenkins to be used

They specify different build environments for Jenkins jobs

They set up network policies for Jenkins

Which of the following is a key benefit of the approach outlined in this chapter?

Reduction in the number of required Jenkins plugins

Ability to bypass Kubernetes for certain builds

Enhanced scalability and efficient resource utilization

Automatic generation of source code

How does this chapter suggest handling the Jenkins setup for testing and demonstration purposes?

By using a complex, multi-stage pipeline

By employing a straightforward inline pipeline

Through external third-party testing services

By manual testing without automation

Answers

b

b

c

c

b

Questions

How does the choice of external storage impact the scalability and reliability of a Jenkins setup in Kubernetes environments?

In what ways can the dynamic creation of pods for Jenkins jobs, as facilitated by the Jenkins Kubernetes plugin, be optimized for different types of CI/CD workflows?

How does the integration of Jenkins with Kubernetes compare with traditional CI/CD pipeline setups in terms of efficiency, cost, and ease of management?

What are the implications of the setup described in this chapter for the security and compliance aspects of a CI/CD pipeline?

Key Terms

Jenkins Kubernetes Plugin: A crucial tool that allows Jenkins to dynamically create and manage pods within a Kubernetes cluster for running build jobs.

External Storage: The use of persistent storage solutions outside of Jenkins to ensure data persistence and scalability within the Kubernetes environment.

Scalable Jenkins Environment: A Jenkins setup configured to efficiently handle varying workloads and scale resources dynamically, particularly within a Kubernetes cluster.

CHAPTER 20

GitOps using ArgoCD and GitHub

Introduction

In the last chapter, we set up a scalable Jenkins deployment on Kubernetes, emphasizing external storage and Kubernetes pods for Jenkins jobs for scalability. It covered the Jenkins Kubernetes plugin for dynamic resource allocation and efficient execution, and provided a practical example with an inline pipeline.

In this chapter, we will use GitOps - a cutting-edge methodology transforming DevOps practices by integrating Git with operations for enhanced efficiency and control. This chapter explores the synergistic use of GitHub, a popular version control platform, and ArgoCD, a Kubernetes-native continuous delivery tool, to implement GitOps. By leveraging these technologies, we demonstrate how to create a more streamlined, transparent, and automated deployment process in Kubernetes environments.

This chapter uses a KIND based kubernetes cluster. You can delete earlier created clusters if any, create a new KIND cluster having minimum 2 nodes before running examples referred in this chapter.

Structure

In this chapter, the following topics will be covered:

Overview

GitOps Setup

Overview

GitOps is a paradigm for managing infrastructure and applications by using Git as the source of truth and automation tool. It integrates development and operations processes, where all changes are committed to Git, ensuring consistency and traceability. This approach emphasizes declarative configuration, automated deployment, and continuous monitoring, facilitating efficient and secure management of cloud-native applications. GitOps streamlines deployment, scaling, and recovery by treating infrastructure and application configuration as code.

This chapter provides a step-by-step guide on how to implement GitOps using an example.

In this chapter, the following two GitHub repos are used as part of the demo. To replicate the demo at your side, you need to fork the repo and replace the GitHub id with your own.

Main Repo gitops-k8s-apps (https://github.com/brainupgrade-in/gitops-k8s-apps/tree/main):

This is the main repo containing a single source of truth. Applications to be managed in the kubernetes cluster are specified here- https://github.com/brainupgrade-in/gitops-k8s-apps/blob/main/argocd/kustomization.yaml (refer to Figure 20.1). Our focus is on the applications highlighted using green colored box.

Figure 20.1: Apps in the Kubernetes cluster

Repo gitops-apps-hello (https://github.com/brainupgrade-in/gitops-apps-hello/tree/main) :

This repo corresponds to one of the apps running in the kubernetes cluster.

GitOps - Setup

ArgoCD

Clone the following repo and run the subsequent commands.

git clone https://github.com/brainupgrade-in/gitops-k8s-apps

cd gitops-k8s-apps/argocd/

kubectl create ns argocd

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

kubectl apply -k .

This will result in ArgoCD syncing all apps defined in ./argocd/kustomization.yaml in the current kubernetes cluster.

Obtain password and access ArgoCD Dashboard using the following commands. Username is admin.

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath=”{.data.password}” | base64 -d; echo

kubectl port-forward -n argocd svc/argocd-server 9999:443

Within a few minutes, ArgoCD will sync up with the main GitHub repo and ensure that all the apps are synced all the time. If any of the apps is not found in the cluster, then it will be deployed (refer to Figure 20.2).

Figure 20.2: ArgoCD Dashboard

Jenkins

As instructed in Chapter 19, Scalable Jenkins on Kuberetes, install the Jenkins Server.

This app pipeline (https://github.com/brainupgrade-in/gitops-apps-hello/blob/main/Jenkinsfile) requires Github and Docker credentials, hence add those using Manage Jenkins Credentials settings (h ) feature (refer to Figure 20.3):

Figure 20.3: Jenkins credentials (Git and Dockerhub)

This jenkins job is using docker service to build images. Launch docker service using the following commands.

kubectl create service clusterip docker --tcp=2375:2375 -n jenkins

kubectl set selector svc docker app=jenkins -n jenkins

Now, let us set up a Multibranch Pipeline job named hello-app-multi-branch for the hello app repo (h) and add the following info, as shown in Figure 20.4:

Figure 20.4: Pipeline script from SCM - Hello app

The Jenkinsfile (https://github.com/brainupgrade-in/gitops-apps-hello/blob/main/Jenkinsfile) referred in the above job contains code to build the docker image with the change commit id as its tag, push it to docker hub, wait for user input and if it is positive then updates the gitops-k8s-apps repo (single source of truth for the apps deployed in the cluster) with the image details that includes commit id.

Once you save this job, it automatically scans for all the pull requests and launches jobs for each open pull request. Also, if there are new changes committed to the main branch, it creates a job for the main branch too. If your Jenkins is accessible from Github then you can also configure automated builds using GitHub Pull Request Builder / GitHub Webhooks.

If the job is running on the main branch, then it will wait for approval before proceeding to deploy the app in the e2e environment. And if approved, it updates the main repo, gitops-k8s-apps, with the latest commit id as the image tag for this app (refer to Figures 20.5–20.7)

Figure 20.5: Jenkins job - E2E deployment awaiting approval

Once the user clicks Proceed, the jenkins job updates the main repo, gitops-k8s-apps, with this commit id for the E2E environment (refer to Figure 20.6):

Figure 20.6: Build Job - post approval

Figure 20.7: Main repo updated with commit id (e2e)

The moment, main repo gitops-k8s-apps, is updated the ArgoCD ensures that this latest image tag is synced with the deployed application in kubernetes cluster.

In essence, ArgoCD ensures that the apps in the kubernetes cluster are synced with the definition available on the GitHub main repo (single source of truth). Jenkins on the other hand manages automated build and deployment via updating the image details in the main repo as part of this pipeline.

Conclusion

This chapter demonstrated the practical implementation of GitOps using GitHub and ArgoCD in a Kubernetes environment. By integrating Jenkins for continuous integration and ArgoCD for continuous deployment, we establish a robust, automated CI/CD pipeline. The pipeline ensures that Kubernetes applications are consistently deployed and synced with the defined state in GitHub. This approach exemplifies an efficient, transparent, and controlled application management strategy in cloud-native ecosystems, showcasing the synergy between GitOps principles and modern DevOps practices.

In the next chapter, we will look at efficient ways to prepare for CKA certification exam.

Points to Remember

Integration of Tools: Understand the synergy between Jenkins for CI, ArgoCD for CD, and GitHub for version control to effectively implement GitOps in Kubernetes.

Automation and Control: Emphasize the importance of automated pipelines for consistent deployment, while incorporating manual approval steps for control and security.

Single Source of Truth: Recognize the critical role of GitHub as the single source of truth in the GitOps model, ensuring consistency and traceability of all deployments.

Multiple Choice Questions

What is the primary role of ArgoCD in GitOps?

Continuous Integration

Continuous Deployment

Monitoring

In a GitOps workflow, what is considered the ‘source of truth’ for application and infrastructure state?

Kubernetes Cluster

Git Repository

Docker Images

Which tool is used for CI in a GitOps pipeline that includes Jenkins, ArgoCD, and GitHub?

Jenkins

ArgoCD

GitHub

Which of the following is essential for ensuring security and control in a GitOps workflow?

Automated pipelines

Manual approval steps

Frequent code changes

What does Kubernetes primarily manage in a GitOps setup?

Version control

Source code

Container orchestration

Answers

b

b

a

b

c

Questions

How can the integration of Jenkins and ArgoCD in a GitOps framework enhance the CI/CD pipeline, especially in terms of efficiency and control?

What challenges might arise when implementing GitOps using GitHub and ArgoCD, and how can they be effectively addressed?

In what ways does treating GitHub as the ‘single source of truth’ impact the overall reliability and scalability of Kubernetes deployments?

Considering the evolving landscape of DevOps, how might the principles and practices of GitOps evolve in the near future to accommodate new technological advancements or organizational needs?

Key Terms

GitOps: A methodology that uses Git as the core tool for managing and automating the provisioning and management of infrastructure and applications.

ArgoCD: A Kubernetes-native continuous delivery tool, utilized within the GitOps framework for deploying applications based on configurations stored in Git repositories.

Continuous Integration/Continuous Deployment (CI/CD): A practice in software development where code changes are automatically tested and deployed, playing a key role in the GitOps pipeline for ensuring reliable and efficient application updates.

CHAPTER 21

CKA Exam Mastery

Introduction

In the last chapter, we explored GitOps - a cutting-edge methodology transforming DevOps practices by integrating Git with operations for enhanced efficiency and control. GitOps is a modern infrastructure management approach that uses Git as a single source of truth for declarative infrastructure and applications. We implemented GitOps using Git and ArgoCD, integrated with Jenkins for continuous integration, establishing Git as the single source of truth.

This chapter succinctly encapsulates effective strategies and insights for excelling in the Certified Kubernetes Administrator exam. It emphasizes the importance of a comprehensive understanding of Kubernetes, combined with practical, hands-on experience. This chapter underlines a targeted approach towards achieving proficiency and success in one of the most respected certifications in cloud-native technologies.

Structure

In this chapter, the following topics will be covered:

Overview

Exam Areas

Format

Scenario-based Tasks

Tips for Success

Overview

A well-structured study plan and specific tips for the Certified Kubernetes Administrator (CKA) exam are essential due to its broad and complex scope, focusing on a range of Kubernetes topics. The practical, hands-on nature of the CKA exam necessitates a preparation approach that emphasizes real-world skills alongside theoretical understanding. Effective time management during preparation and understanding the unique performance-based format of the exam are critical. These strategies ensure a comprehensive and efficient approach to mastering the necessary skills and knowledge for success in the CKA exam.

Exam Areas

In the Certified Kubernetes Administrator (CKA) exam, certain areas have higher weightage and are crucial for a successful outcome. Based on the latest exam curriculum, here are the key domains and their approximate weightages (source: https://github.com/cncf/curriculum/blob/master/CKA_Curriculum_v1.28.pdf):

Cluster Architecture, Installation, and Configuration (25%)

Manage Role-Based Access Control(RBAC)

Use Kubeadm to install a basic cluster

Manage a highly-available Kubernetes cluster

Provision underlying infrastructure to deploy a Kubernetes cluster

Perform a version upgrade on a Kubernetes cluster using Kubeadm

Implement etcd backup and restore

Workloads and Scheduling (15%)

Understand deployments and how to perform rolling update and rollbacks

Use ConfigMaps and Secrets to configure applications

Know how to scale applications

Understand the primitives used to create robust, self-healing, application deployments

Understand how resource limits can affect Pod scheduling

Awareness of manifest management and common templating tools

Services and Networking (20%)

Understand host networking configuration on the cluster nodes

Understand connectivity between Pods

Understand ClusterIP, NodePort, LoadBalancer service types and endpoints

Know how to use Ingress controllers and Ingress resources

Know how to configure and use CoreDNS

Choose an appropriate container network interface plugin

Storage (10%)

Understand storage classes, persistent volumes

Understand volume mode, access modes and reclaim policies for volumes

Understand persistent volume claims primitive

Know how to configure applications with persistent storage

Troubleshooting (30%)

Evaluate cluster and node logging

Understand how to monitor applications

Manage container stdout and stderr logs

Troubleshoot application failure

Troubleshoot cluster component failure

Troubleshoot networking

To perform well in the CKA exam, focus on these high-weighting areas, but don’t neglect the other topics. Regular practice, combined with a strong theoretical understanding, will be key to mastering these concepts.

Format

Type of Exam: Performance based

Duration: 2 hours

Environment:

You will be given access to a remote linux desktop in a browser-based environment.

You can open and use the official Kubernetes documentation during the exam for reference.

The environment is typically a set of one or more Kubernetes clusters where you’ll perform the tasks.

Number of Questions: Approximately 15-20 tasks or problems to solve.

Passing Score: Typically, around 66%.

Scenario-Based Tasks

You’ll face real-world scenarios requiring you to use your Kubernetes knowledge to solve specific problems. Tasks vary in complexity, from simple commands to more complex configurations.

Examples might include:

Setting up a Kubernetes cluster.

Deploying an application on Kubernetes.

Configuring network policies.

Managing storage solutions.

Troubleshooting and resolving issues with the cluster or applications running on it.

Implementing security measures such as RBAC.

Tips for Success

To effectively prepare for the Certified Kubernetes Administrator (CKA) exam, focus on a blend of theoretical knowledge and practical experience. Here’s a structured approach:

Understand the Exam Blueprint: Familiarize yourself with the exam format and content areas. The CKA exam is heavily practical, focusing on tasks related to setting up and managing Kubernetes clusters.

Build a Strong Foundation: Ensure a solid grasp of Kubernetes fundamentals. Understand core concepts such as Pods, Deployments, Services, Networking, Persistent Storage, and Security.

Hands-On Practice: Set up your Kubernetes clusters to practice. Use minikube or KIND for local experimentation. Focus on deploying applications, troubleshooting, cluster maintenance, and security.

Command Line Proficiency: Gain expertise in using kubectl, Kubernetes’ command-line tool. The exam requires performing tasks directly using command-line utilities.

Leverage Official Documentation: Get comfortable with the Kubernetes official documentation. You can access it during the exam, so knowing how to quickly find information is crucial.

Time Management Skills: Develop efficiency in solving problems, as the exam is time-constrained. Practice doing tasks under timed conditions.

Take Practice Exams: Engage in mock exams to familiarize yourself with the exam environment and type of questions.

Study Groups and Forums: Participate in community forums, study groups, and Kubernetes meetups. Learning from peers can offer new insights and tips.

Stay Updated: Keep up with the latest Kubernetes updates and changes, as the exam content evolves over time.

Relax and Focus During the Exam: Read each question carefully and plan your approach before executing. Time management is key.

Remember, CKA is about demonstrating practical skills in a real environment. Continuous hands-on practice and familiarity with Kubernetes documentation are crucial for success.

Conclusion

This chapter provided a focused guide for the Certified Kubernetes Administrator (CKA) exam. We delved into the exam areas, format and scenario of the CKA exam, underlining its practical, hands-on nature and the critical areas of focus such as cluster architecture, workloads, services, networking, and troubleshooting.

Points to Remember

Understand Kubernetes Architecture: Be thoroughly familiar with the components such as Nodes, Pods, Services, and how they interact.

Hands-On Practice: Regularly practice real-world scenarios, as the exam is heavily practical.

Master kubectl: Proficiency with kubectl commands is vital for performing tasks efficiently during the exam.

Familiarize with the Exam Environment: Understand the format, the type of questions, and the interface you’ll be working with.

Efficient Use of Documentation: Knowing how to quickly find information in the official Kubernetes documentation is crucial.

Time Management Skills: Learn to allocate your time wisely across different questions.

Troubleshooting: Be adept at diagnosing and resolving common issues in Kubernetes environments.

Networking and Security: Have a strong grasp of Kubernetes networking and security configurations.

Cluster Maintenance: Understand cluster upgrading, backup, and recovery processes.

Stay Updated: Keep abreast of the latest Kubernetes features and best practices, as the platform evolves rapidly.

Multiple Choice Questions

What does the ‘kubectl’ command-line tool primarily interact with in a Kubernetes cluster?

Docker Engine

Kubelet

API Server

etcd

Which of the following is a Kubernetes controller that ensures the correct number of pod replicas are running?

Scheduler

ReplicaSet

Deployment

StatefulSet

Which Kubernetes object is used to expose a set of pods to the network?

Deployment

Service

Volume

ReplicaSet

What is the purpose of a Namespace in Kubernetes?

To isolate network traffic

To manage resource quotas

To provide storage

To run privileged pods

What is a ‘Node’ in Kubernetes?

A pod running on a cluster

A container running within a pod

A machine, VM, or physical server

A storage resource

Which resource in Kubernetes is used to manage external access to the services in a cluster, typically HTTP?

Service

Ingress

NodePort

ClusterIP

Which of the following is a key component of Kubernetes that stores the cluster’s configuration data?

Scheduler

Kube-proxy

etcd

Controller Manager

In Kubernetes, what is a pod?

A group of nodes

The smallest deployable unit

A storage unit

A networking rule

Which command is used to scale a deployment in Kubernetes?

kubectl scale

kubectl set

kubectl update

kubectl expose

What does a Kubernetes ‘Service’ do?

Manages a set of pods

Exposes an application running on a set of pods

Stores data persistently across the cluster

Automates container deployment

What is the role of ‘Kubelet’ in a Kubernetes cluster?

Manages the pods

Schedules pods

Stores configuration data

Balances network load

Which Kubernetes component is responsible for distributing network traffic to various pods?

Ingress Controller

Kube-proxy

Service

Endpoint

What is the main role of a Kubernetes ‘Scheduler’?

To replicate pods

To manage storage

To assign pods to nodes

To expose services

In Kubernetes, what is RBAC primarily used for?

Networking

Scheduling

Authorization

Storage management

What type of storage in Kubernetes allows multiple nodes to access the same storage system?

Local

Ephemeral

Persistent

Shared

In Kubernetes, what does a ‘StatefulSet’ resource provide?

Stateless deployment

Load balancing

Persistent storage

Single pod per node

Which Kubernetes feature ensures that the pods are only scheduled on nodes with specific labels?

Node affinity

Taints and tolerations

Node selectors

Resource quotas

What is the purpose of a ‘DaemonSet’ in Kubernetes?

To run a single pod on each node

To manage storage volumes

To expose services externally

To balance load across pods

Which of these is not a valid Kubernetes volume type?

NFS

ConfigMap

Secret

S3Bucket

In Kubernetes, what is the primary role of ‘etcd’?

Load balancing

Service discovery

Configuration data storage

Scheduling pods

Which command in Kubernetes is used to get logs from a specific pod?

kubectl describe pod

kubectl logs

kubectl get pod

kubectl exec

What is the default scheduler policy in Kubernetes?

Round-robin

Random

Least resource usage

Spread across zones

Which Kubernetes object is immutable once created?

Pod

Service

ConfigMap

Secret

What is the Kubernetes command to view the current configuration of your cluster?

kubectl config view

kubectl cluster-info

kubectl get nodes

kubectl describe cluster

In Kubernetes, how are updates to pods in a deployment managed?

Rolling Update

Blue/Green Deployment

Canary Release

All of the above

What is Helm in Kubernetes?

A package manager for Kubernetes applications

A tool for automatic scaling

A cluster monitoring tool

A network policy editor

Answers

c

b

b

b

c

b

c

b

a

b

a

b

c

c

d

c

c

a

d

c

b

c

a

a

d

a

Questions

What are the key components of a Kubernetes cluster, and how do they interact?

How do you deploy and manage applications using Kubernetes?

What are the best practices for Kubernetes networking and security?

How do you perform cluster maintenance and troubleshooting?

How are storage and data persistence managed in Kubernetes?

Key Terms

Pod: The smallest deployable unit in Kubernetes that represents a set of running containers on your cluster.

Deployment: A higher-level concept that manages declarative updates for pods and ReplicaSets.

Node: A worker machine in Kubernetes, which may be a VM or physical machine, depending on the cluster.

Cluster: A set of node machines for running containerized applications managed by Kubernetes.

kubectl: The command-line tool for interacting with the Kubernetes cluster.

etcd: A consistent and highly-available key value store used as Kubernetes’ backing store for all cluster data.

Service: An abstraction that defines a logical set of pods and a policy by which to access them.

Ingress: Forwards external requests to the services running inside the cluster. Requests are generally HTTP/HTTPS.

ConfigMap: This Kubernetes object holds a key-value pair that can be used by pod.

Secret: An object that contains a small amount of sensitive data such as a password, a token, or a key.

Namespace: A way to divide cluster resources between multiple users.

Persistent Volume (PV): Represents a storage created manually or dynamically using storage classes.

Persistent Volume Claim (PVC): A request for storage by a user.

DaemonSet: Ensures that all (or some) nodes run a copy of a pod.

Role-Based Access Control (RBAC): A method of regulating access to computer or network resources based on the roles of individual users within an enterprise.

Scheduler: Component in Kubernetes that assigns newly created pods to nodes.

Kube-proxy: Network proxy that runs on each node in the cluster, implementing part of the Kubernetes Service concept.

StatefulSet: Manages the deployment and scaling of a set of pods, and provides guarantees about the ordering and uniqueness of these pods.

ReplicaSet: This controller ensures that desired replicas are running.

Taints and Tolerations: Node affinity in Kubernetes is controlled by taints (on nodes) and tolerations (on pods).

Index

A

API Server

about 200

admission, controlling 201

authentication, authorizing 200

Pod Security Standards 201

API Server Authentication, mechanisms

basic, authentication 200

bearer, tokens 200

client, certificates 200

OIDC 200

API Server Authorization, mechanisms

ABAC 200

node, authorization 200

RBAC 200

Webhook 200

API Server Controllers, types

admission, mutating 201

admission, validating 201

APM Server

about 272

installing 272

Metrics 276

Weather Application 273

APM Server Microservices, types

API Service 273

database 273

web frontend 273

Audit Logging

about 210

API Server, configuring 210

backend, logging 210

behavior, configuring 210

event, recording 210

file rules, optimizing 211

log, analyzing 210

B

Blue Green 84

C

Calico CNI Plugin 189

Calico CNI Plugin, components

BGP 190

control plane 190

data plane 190

felix 190

typha 190

Calico CNI Plugin, features

cross-subnet, networking 190

high, performance 190

IPv4/IPv6, supporting 190

network, enforcement 190

scalability 190

Certificate Manager

about 260

configuring 260, 261

installing 260

verifying 261

Certified Kubernetes Administrator (CKA)

about 300

format, utilizing 302

key, domains 301, 302

real-world, scenarios 302

success, tips 303

ClusterIP 176

cluster.yaml 72, 84

CNI Plugins, aspects

calico 185

canal 186

cilium 186

flannel 185

weave 186

CNI Plugins, criteria

community, supporting 186

compatibility 186

ease, use 186

feature, sets 186

network, requirements 186

CNI Plugins, installing 186, 187

ConfigMap 109

Container Probes, phase

livenessProbe 33

readinessProbe 32

startupProbe 31, 32

control plane, components

API, controlling 5

Cloud Controller Manager (CCM) 6

ETCD, database 5

Kube Controller Manager (KCM) 5

scheduler 5

CoreDNS

about 157

architecture 158

deployment, configuring 159

diagnostic tools, utilizing 165

key aspects, interacting 163

Kubernetes, integrating 158

performance, scalability 163

Pods, configuring 162

security 164

CoreDNS, challenges

data, incorrecting 165

DNS, resoluting 165

functionality, utilizing 165

high, latency 165

CoreDNS, components

core binary 158

Corefile 158

load, balancing 158

mechanisms, service 158

plugins 158

CoreDNS, concepts

extensibility 157

flexibility 157

performance 157

simplicity 157

CoreDNS, customizing options

DNS Entries, adding 161

DNS Services, integrating 161

forwarding 161

load, balancing 161

CoreDNS, key features

CRDs 159

network, policies 159

Pod DNS, resoluting 159

reliability, scalability 159

service, discovery 158

CoreDNS Performance, considering

availability, tuning 164

Corefile, configuring 163

load, balancing 164

resources, allocating 164

strategies, caching 164

CoreDNS, scaling types

autoscale, configuring 164

cluster size, considering 164

horizontal 164

CoreDNS Security, keys

network, policies 164

Pod Security, policies 165

query monitor, analyzing 165

Restricted Access, controlling 164

services, encrypting 165

update, patching 165

Corefile 160

Corefile Plugins, types

cache 161

forward 161

health 161

Kubernetes 161

log 161

metrics 161

Corefile, structures

directives 161

plugins 161

zone, blocks 160

D

Docker 13

E

emptyDir 85

ETCD

about 130

clusters, scaling 139

data manipulate, accessing 134

importance 134

Kubernetes Cluster, managing 130

security 139

ETCD, algorithm consensus

data, consistency 131

Leader, election 131

Raft, basic 130

ETCD, aspects

API Server, integrating 132

data, directory 132

Time-To-Live (TTL) 132

ETCD Backups, best practices

backup, validating 139

frequent, backup 138

offsite, storage 138

etcdctl 134, 135

etcdctl, operations

health, checking 136

keys, listing 136

ETCD, features

availability 130

consistency 130

reliability 130

ETCD, keys

hierarchy, key 132

mechanism, versioning 132

ETCD Kubernetes, data essential

ConfigMaps, secrets 131

ingress rules, policies 131

pods 131

services 131

ETCD Kubernetes, operations

consistency, guarantees 131

fault, tolerance 131

ETCD Kubernetes, uses

cluster, coordinating 131

cluster state, storage 131

data resource, configuring 131

ETCD Restoring, commands

backing up 136

data, restoration 137

ETCD, setting up

initial, configuring 132

install, options 132

network, configuring 132

etcd.yaml 138

External-to-Service Communication 177, 178

F

Flannel CNI Plugin 192

Flannel CNI Plugin, features

backend, supporting 192

Kubernetes, integrating 192

network, isolating 192

overlay, networking 192

simple, configuring 192

Flannel CNI Plugin, use cases

CI/CD Pipelines 193

environments, testing 192

experiment, learning 193

network, requirements 193

small clusters 192

G

GitOps 291, 292

GitOps, setup 292-297

H

hostPath 86, 87

I

Ingress

about 50

App URL, testing 53

components, interacting 53

controller, installing 50

Hello App, running 50, 51

Ingress Controller

about 260

installing 260

verifying 260

Ingress-to-Service Communication 178-180

K

KCM, processes

endpoint slice, controlling 6

job, controlling 5

node, controlling 5

service account, controlling 6

KIND Kubernetes

about 16, 17

cluster, testing 17

Kind Cluster, cleaning 18

multi-node, utilizing 18

kubeadm 144

Kubeadm

about 19

clean, up 24

cluster nodes, launching 19-21

control plane, interacting 23, 24

install, ways 19

Master/Control, initializing 21-23

kubelet 148

Kubelet

about 188

network, policy 190

Pod Network, key steps 188, 189

Kubernetes

about 1, 2

capacity, deploying 60-62

components 58

Horizontal Pod Autoscaler (HPA) 64-67

metrics server, utilizing 58, 59

Network Policy 205

scalability 62-64

Kubernetes, benefits

Automatic Bin, packing 3

Rollouts/Rollbacks, automating 2, 3

secrets configure, managing 3, 4

self-healing 3

service load, balancing 2

storage, orchestration 3

Kubernetes Cluster, commands

cluster, launching 146-149

KIND Utility, installing 145

worker nodes, upgrading 149-151

Kubernetes, clustering methods

Oracle VirtualBox 11

vagrant 11

Kubernetes CNI Model 185

Kubernetes Deployment, strategies

Blue Green (aka Red Black) 75-78

canary 79, 80

Recreate 75

RollingUpdate 73, 74

Kubernetes Network Model 169, 170

Kubernetes Network Model, characteristics

External-to-Service, communicating 170

Pod, communicating 170

Pod-to-Pod, communicating 170

Pod-to-Service, communicating 170

Kubernetes, perspectives views

Dynamic View 6, 7

Logical View 4

Kubernetes Probes, types

livenessProbe 121, 122

readinessProbe 119, 120

startupProbe 117-119

Kubernetes Security Model 199

Kubernetes Security Model, mechanisms

admission, controlling 199

audit, logging 199

authentication 199

authorization 199

ETCD, roles 199

network, policies 199

secrets, managing 199

TLS, communicating 199

Kubernetes, tools

Calico 258, 259

Certificate Manager 260

CNI Plugin 258

EFK 263

Elastic APM 263

Grafana 262

Ingress Controller 259

Metrics Server 259

Prometheus 262

L

livenessProbe, configuring

failure, tolerance 122

initial, delay 122

probe, interval 122

probe, machanism 122

success, criteria 122

Logical View, components

control plane 5

worker plane 6

M

Metrics

about 276

application, optimizing 277

cluster, utilizing 277

Prometheus/Grafana, installing 276

Metrics Server

about 259

cluster, installing 259

verifying 259

Microk8s

about 15

cluster, cleaning 16

Kubernetes, testing 16

Minikube

about 12

cleaning 15

cluster, testing 14, 15

Docker 13

installing 14

Kubernetes Cluster, launching 14

Ubuntu, launching 12, 13

N

Network Policy

about 205

concepts, analyzing 206-208

payroll app, communicating 206

scenario, utilizing 206

Nginx, setting up 51-53

NodePort Service, running 47, 48

O

Observability 269

Observability, fundamentals

logs, event 269

metrics 269

tracing 269

Observability, tools

APM Server 269

Elasticsearch 269

Fluent-bit 269

Grafana 269

Kibana 269

Prometheus 269

Oracle VirtualBox

reference link 11

P

Persistent Volume 87

PersistentVolumeClaim

about 87, 88

AWS EBS, obtaining 90

modes, accessing 89

storage class, utilizing 89, 90

types 89

Persistent Volume, storage type

ReadOnlyMany (ROX) 89

ReadWriteMany (RWX) 89

ReadWriteOncePod (RWOP) 89

ReadWriteOnce (RWO) 89

Pod

about 28

command, imperamative 29, 30

Container Probes 31

lifecycle, utilizing 31

resources, deploying 33-35

restart, policy 30

Rollback 36, 37

Rollout, changing 36

scalability 35

Pod Communication 171-173

Pod Lifecycle, phase

failed 31

pending 31

running 31

succeeded 31

unknown 31

Pod Security Standards, modes

audit 202

enforce 202

warn 202

Pod Security Standards, profiles

baseline 202

privileged 201

restricted 202

Pod Storage 85, 86

Pod-to-Pod Communication 173, 174

Pod-to-Service Communication 176, 177

Probe, mechanisms

Exec 123

gRPC 123

HTTP GET 122

Probes 117

R

RBAC (Role-Based Access Control)

about 202

implementing, steps 203, 204

Jenkins CICD, deploying 205

principle, roles 203

readinessProbe, configuring

fault, tolerance 120

initial delay 120

probe, frequency 120

probe type 120

success, criterion 120

Recreate 84

Redis App, scaling 100, 101

Redis Database, use case 97-100

Release Cycle 145

Release Cycle, features

backup, strategies 145

data sefety, ensuring 145

ETCD Data, backing 145

ReplicaSet 34

Rollback Troubleshoot, cases

backups, restoring 152

components, downgrading 152

issues/tips, upgrading 152

resource limitations, dealing 152

strategies, upgrading 152

version, handling 152

RollingUpdate, strategy

maxSurge 73

maxUnavailable 73

S

Scalable Jenkins

about 283

access, controlling 284, 285

installing 283, 284

Kubernetes Plugin, setup 285-287

secret object 114-117

Service 43-46

Service, types

ClusterIP 46

ExternalName 49

LoadBalancer 48, 49

NodePort 47

startupProbe, configuring

check, frequency 118

failure, tolerance 118

initial delay 118

probe type 118

success, criteria 118

StatefulSet 43

about 37

limitations 37

postgre database, running 37-39

StatefulSet, key features

automate roll, updating 97

greaceful deploy, scaling 97

network identify, stabling 97

parallel pod, managing 97

persistent storage, stabling 97

StatefulSet, limitations 103

StatefulSet Pod, policies

partitioned, rolling 103

roll, updating 101, 102

T

TLS, benefits

authenticating 209

data, integrity 209

encrypting 209

Transport Layer Security (TLS) 209

Troubleshoot 216

Troubleshoot, scenarios

application resource, optimizing 229, 230

blue-green deploy, implementing 249, 251

Cluster Monitor, setting up 228, 229

Cluster, restoring 233, 234

Cluster-Wide, logging 240, 241

ConfigMap Secret, managing 221, 222

ingress service, troubleshooting 218, 219

Kubernetes Cluster, upgrading 227, 228

metrics, auto scaling 234, 235

namespace, configuring 247-249

network implement, testing 230-232

network issues, debugging 244, 245

network policy, verifying 216, 217

network service, configuring 224, 225

node issues, investigating 218

persistent storage, configuring 245-247

persist volume, troubleshooting 219-221

RBAC, configuring 222-224

resource quota, managing 217

sensitive data, securing 225-227

SSL Termination, configuring 241-244

techniques, implementing 237-239

Transit Data, securing 235-237

V

vagrant

reference link 11

W

Weather Application 273-275

Weave Net CNI Plugin 193

Weave Net CNI Plugin, features

ease use 193

fault, tolerance 193

network, encrypting 193

overlay, networking 193

policies, supporting 193

Weave Net CNI Plugin, use cases

development, testing 194

dynamic, environments 194

environments, securing 193

high availability, setup 194

multi-host, clusters 193

worker plane, types

Kubelet 6

Kube-proxy 6