Day One :

install Proxmox on an OVH server with a public IP and then install and configure Ubuntu as a virtual machine, you can follow these steps:

1. Provision OVH Server: Access your OVH account and provision a server with the desired specifications, including a public IP address.

2. Connect to the Server: Once the server is provisioned, you can connect to it via SSH using a terminal application like Terminal (Mac/Linux). Use the server's public IP address and the SSH credentials provided by OVH.

3. Install Proxmox VE: Proxmox VE is a server virtualization platform that will allow you to create and manage virtual machines. Follow the official Proxmox VE installation guide specific to your server's operating system. Typically, it involves running a script provided by Proxmox to install the necessary packages.

4. Access Proxmox Web Interface: Once Proxmox VE is installed, access its web interface by opening a web browser and entering the server's IP address followed by the default port 8006. For example, https://<server_ip>:8006. Accept any security warnings and log in using the default username root and the password set during the Proxmox installation.

5. Configure Network: In the Proxmox web interface, navigate to "Datacenter" -> "Network" -> "VM Network" and configure the network settings for your virtual machines. Ensure the public IP address assigned to your OVH server is properly configured.

6. Create Virtual Machine (VM): In the Proxmox web interface, click on "Create VM" to create a new virtual machine. Follow the wizard to specify the VM settings, such as the amount of RAM, CPU cores, disk space, and networking. Choose Ubuntu as the guest OS template.

7. Install Ubuntu on the VM: After creating the VM, select it in the Proxmox interface and click on "Console" to access the virtual machine's console. Start the VM and mount the Ubuntu ISO image to perform the installation. Follow the Ubuntu installation process, configuring options like language, partitioning, and user credentials.

8. Configure Ubuntu: Once Ubuntu is installed, log in to the virtual machine and perform any necessary configurations. Update the system packages using sudo apt update and sudo apt upgrade. Install additional software, set up firewall rules, configure network interfaces, etc., as per your requirements.

9. Access Ubuntu VM: To access the Ubuntu VM from your local machine, you can use SSH or remote desktop tools like VNC. Ensure you have network connectivity between your local machine and the OVH server.

Day TWO :

1. Containerize application: Package your application and its dependencies into a container image. Kubernetes uses containers to run and manage applications. You can use tools like Docker to create container images.

2. Create Kubernetes manifests: Kubernetes uses YAML or JSON files called manifests to define the desired state of your application. Manifests describe the resources that Kubernetes should create and manage, such as pods, services, deployments, and ingress rules. You need to create these manifests to define your application's structure, dependencies, and configurations.

3. Define a Deployment: A Deployment is a Kubernetes resource that manages the lifecycle of your application. It ensures that the desired number of instances (replicas) of your application are running and handles updates and rollbacks. You define a Deployment in your manifest file, specifying details like the container image, replicas, labels, and environment variables.

4. Apply the manifests: Use the kubectl apply command to apply your Kubernetes manifests to the cluster. This command creates or updates the specified resources based on the desired state defined in the manifests.

5. Verify the deployment: After applying the manifests, you can use kubectl commands to verify the status of your deployment. For example, kubectl get pods shows the running pods, kubectl get deployments shows the status of the deployments, and kubectl logs <pod-name> retrieves the logs of a specific pod.

6. Expose the application: To make your application accessible from outside the cluster, you'll need to expose it using a Service. A Service is a Kubernetes resource that provides a stable network endpoint to access your application. There are different types of Services, such as ClusterIP, NodePort, and LoadBalancer. You define and create a Service in your manifest file.

7. Scale and update the deployment: Kubernetes allows you to scale your application horizontally by adjusting the number of replicas in your Deployment. You can use the kubectl scale command to scale your application. To update your application, you modify the Deployment's manifest file with the new version or configuration and apply the changes using kubectl apply again.

8. Monitor and manage: Kubernetes provides various tools and approaches for monitoring and managing your application. You can use kubectl commands, Kubernetes Dashboard, or third-party monitoring tools to monitor the health, resource usage, and logs of your application.

Day Three :

build a setup that includes Amazon S3 for Laravel media storage, Amazon CloudFront for content delivery, and Amazon SES for email integration in a Laravel application, you can follow these steps:

2. Configure Laravel: Install and configure Laravel for your application. This involves setting up the database connection, configuring the mail driver (to use SES later), and any other necessary Laravel configurations.

3. Set up Amazon S3: Create an S3 bucket in the AWS Management Console. This bucket will store your media files. Ensure you have the appropriate permissions to access the bucket. You may need to create an IAM user and attach the necessary policies.

4. Install and Configure Laravel S3 Driver: Install the league/flysystem-aws-s3-v3 package through Composer to enable Laravel's S3 driver. Configure the S3 driver in Laravel's filesystems.php configuration file, specifying the S3 bucket and credentials.

5. Upload and Retrieve Media: In your Laravel application, use the storage facade to upload and retrieve media files. For example, you can use Storage::put() to upload files to S3 and Storage::url() to generate URLs for accessing the files.

6. Set up Amazon CloudFront: Create a CloudFront distribution in the AWS Management Console. Configure CloudFront to use your S3 bucket as the origin and specify the desired settings, such as caching, SSL, and custom domain names. CloudFront acts as a content delivery network (CDN) to cache and deliver your media files globally.

7. Integrate CloudFront URLs in Laravel: Replace the direct S3 URLs in your Laravel application with the CloudFront URLs generated for your media files. This ensures that files are served through CloudFront for faster and more efficient delivery.

8. Set up Amazon SES: In the AWS Management Console, create an SES (Simple Email Service) configuration, verify your domain or email addresses, and configure the necessary email settings, such as DKIM and SPF records.

9. Configure Laravel Mail: Update Laravel's mail driver configuration to use SES. Modify the mail.php configuration file and specify the SES SMTP credentials and region.

10. Send Emails: Use Laravel's built-in mail functionality (Mail facade) to send emails from your application. Laravel will use SES as the underlying mail transport.

11. Test and Verify: Test the media upload/download functionality and email sending from your Laravel application to ensure everything is working as expected. Monitor logs and error messages to identify and resolve any issues.

Day Four :

1. Connect a GitHub Repository:GitHub reposito y application's source code. Initialize the repository with your project files and push them to the remote repository.

2. Choose a CI/CD Platform: Select a CI/CD platform that integrates well with GitHub. Some popular options include devtron .

3. Configure CI/CD with GitHub Actions: Create a Dockerfile , nginx conf and supervisor.sh directory in your GitHub repository. Inside this directory, create a YAML file (e.g., ci-cd.yaml) to define your CI/CD workflow. Configure the workflow to trigger on specific events, such as a push to the master branch or a pull request. Define the necessary steps, such as building the application, running tests, and creating a build artifact.

4. Define Environment-Specific Configurations: Determine the necessary configurations for your staging and production environments. This may include environment variables, database connections, and other settings specifi

5. Set up Staging Environment: Create a staging environment using Devtron or any other Kubernetes deployment tool of your choice. Devtron simplifies deployment on Kubernetes by providing a user-friendly interface and automation. Configure the necessary Kubernetes resources, such as namespaces, deployments, services, and ingress rules, to create your staging environment.

6. Configure Deployment Workflow: Modify your CI/CD workflow to include the deployment of your application to the staging environment. This involves building a Docker image, pushing it to a container registry like Docker Hub or Amazon ECR, and deploying the image to your staging environment using Devtron or Kubernetes manifest files.

7. Test and Verify Staging Environment: Run tests and perform manual verification to ensure that the staging environment is functioning correctly. This includes testing application functionality, performance, and integration with any dependent services.

8. Set up Production Environment: Create a production environment using Devtron or Kubernetes in a similar manner to the staging environment. Configure the necessary Kubernetes resources according to your production requirements, such as scaling, high availability, and security.

9. Configure Production Deployment Workflow: Modify your CI/CD workflow to include the deployment of your application to the production environment. This may involve additional steps for promoting the application from the staging environment to production, such as manual approval gates or specific branching strategies.

10. Test and Verify Production Environment: Run tests and perform thorough verification to ensure that the production environment is functioning correctly. This includes testing application functionality, security, performance, and any other critical aspects.

11. Monitor and Manage: Implement monitoring and logging solutions to gain insights into your application's performance and health in both staging and production environments. Utilize Devtron's monitoring capabilities or integrate with third-party tools like Prometheus or Grafana.

Day Five :

Build a data protection process for a MySQL database and Laravel media files stored in an S3 bucket with a lifecycle policy without encryption, you can follow these steps:

1. Backup MySQL Database: Implement a regular backup strategy for your MySQL database to protect against data loss. Use tools like mysqldump or database backup services to create automated backups at scheduled intervals. Store the backups in a secure location, such as a separate server or cloud storage.

2. Store Laravel Media Files in S3: Configure your Laravel application to store media files in an S3 bucket. Set up the necessary credentials and access policies to ensure secure access to the bucket. This allows you to offload the storage of media files to a scalable and durable object storage service.

3. Enable Server-Side Logging: Enable server-side logging for the S3 bucket to capture access logs. This will help you monitor and track any unauthorized access attempts or suspicious activities related to your media files.

4. Configure Lifecycle Policy: Define a lifecycle policy for your S3 bucket to manage the lifecycle of the media files. The lifecycle policy can specify rules to transition or expire objects based on criteria such as age, object size, or specific prefixes. For example, you can set rules to automatically move files to Glacier storage after a certain period or delete files that have reached their retention period.

5. Enable Versioning: Enable versioning for the S3 bucket to preserve multiple versions of the media files. This provides an additional layer of protection by allowing you to restore previous versions of files in case of accidental deletion or corruption.

6. Implement Access Controls: Apply appropriate access controls to your MySQL database and the S3 bucket to restrict unauthorized access..

7. Monitor and Test: Regularly monitor the backup process, database integrity, and S3 storage to ensure everything is functioning correctly. Perform periodic tests to restore data from backups and validate the integrity of the restored data.

8. Disaster Recovery Plan: Develop a comprehensive disaster recovery plan that includes steps to restore the MySQL database and media files from backups in case of data loss or system failure. Document the procedures and regularly review and test the plan to ensure its effectiveness.

9. Regularly Update and Patch: Keep your MySQL database, Laravel application, and server infrastructure up to date with the latest security patches and updates. Regularly review and apply Laravel framework updates and security best practices to protect against vulnerabilities.

Moving from physical networks using switches, routers, and ethernet cables to virtual networks using software-defined networks (SDN) and virtual interfaces involves a slight learning curve. Of course, the principles remain the same, but there are different specifications and best practices. Kubernetes has its own set of rules, and if you're dealing with containers and the cloud, it helps to understand how Kubernetes networking works.

The Kubernetes Network Model has a few general rules to keep in mind:

1. Every Pod gets its own IP address: There should be no need to create links between Pods and no need to map container ports to host ports.

2. NAT is not required: Pods on a node should be able to communicate with all Pods on all nodes without NAT.

3. Agents get all-access passes: Agents on a node (system daemons, Kubelet) can communicate with all the Pods in that node.

4. Shared namespaces: Containers within a Pod share a network namespace (IP and MAC address), so they can communicate with each other using the loopback address.

What Kubernetes networking solves

Kubernetes networking is designed to ensure that the different entity types within Kubernetes can communicate. The layout of a Kubernetes infrastructure has, by design, a lot of separation. Namespaces, containers, and Pods are meant to keep components distinct from one another, so a highly structured plan for communication is important.

(Nived Velayudhan, CC BY-SA 4.0)

Container-to-container networking

Container-to-container networking happens through the Pod network namespace. Network namespaces allow you to have separate network interfaces and routing tables that are isolated from the rest of the system and operate independently. Every Pod has its own network namespace, and containers inside that Pod share the same IP address and ports. All communication between these containers happens through localhost, as they are all part of the same namespace. (Represented by the green line in the diagram.)

Pod-to-Pod networking

With Kubernetes, every node has a designated CIDR range of IPs for Pods. This ensures that every Pod receives a unique IP address that other Pods in the cluster can see. When a new Pod is created, the IP addresses never overlap. Unlike container-to-container networking, Pod-to-Pod communication happens using real IPs, whether you deploy the Pod on the same node or a different node in the cluster.

The diagram shows that for Pods to communicate with each other, the traffic must flow between the Pod network namespace and the Root network namespace. This is achieved by connecting both the Pod namespace and the Root namespace by a virtual ethernet device or a veth pair (veth0 to Pod namespace 1 and veth1 to Pod namespace 2 in the diagram). A virtual network bridge connects these virtual interfaces, allowing traffic to flow between them using the Address Resolution Protocol (ARP).

When data is sent from Pod 1 to Pod 2, the flow of events is:

1. Pod 1 traffic flows through eth0 to the Root network namespace's virtual interface veth0.

2. Traffic then goes through veth0 to the virtual bridge, which is connected to veth1.

3. Traffic goes through the virtual bridge to veth1.

4. Finally, traffic reaches the eth0 interface of Pod 2 through veth1.

Pod-to-Service networking

Pods are very dynamic. They may need to scale up or down based on demand. They may be created again in case of an application crash or a node failure. These events cause a Pod's IP address to change, which would make networking a challenge.

Credit to : (Nived Velayudhan, CC BY-SA 4.0)

Kubernetes solves this problem by using the Service function, which does the following:

1. Assigns a static virtual IP address in the frontend to connect any backend Pods associated with the Service.

2. Load-balances any traffic addressed to this virtual IP to the set of backend Pods.

3. Keeps track of the IP address of a Pod, such that even if the Pod IP address changes, the clients don't have any trouble connecting to the Pod because they only directly connect with the static virtual IP address of the Service itself.

The in-cluster load balancing occurs in two ways:

1. IPTABLES: In this mode, kube-proxy watches for changes in the API Server. For each new Service, it installs iptables rules, which capture traffic to the Service's clusterIP and port, then redirects traffic to the backend Pod for the Service. The Pod is selected randomly. This mode is reliable and has a lower system overhead because Linux Netfilter handles traffic without the need to switch between userspace and kernel space.

2. IPVS: IPVS is built on top of Netfilter and implements transport-layer load balancing. IPVS uses the Netfilter hook function, using the hash table as the underlying data structure, and works in the kernel space. This means that kube-proxy in IPVS mode redirects traffic with lower latency, higher throughput, and better performance than kube-proxy in iptables mode.

The diagram above shows the package flow from Pod 1 to Pod 3 through a Service to a different node (marked in red). The package traveling to the virtual bridge would have to use the default route (eth0) as ARP running on the bridge wouldn't understand the Service. Later, the packages have to be filtered by iptables, which uses the rules defined in the node by kube-proxy. Therefore the diagram shows the path as it is.

Internet-to-Service networking

So far, I have discussed how traffic is routed within a cluster. There's another side to Kubernetes networking, though, and that's exposing an application to the external network.

(Nived Velayudhan, CC BY-SA 4.0)

You can expose an application to an external network in two different ways.

1. Egress: Use this when you want to route traffic from your Kubernetes Service out to the Internet. In this case, iptables performs the source NAT, so the traffic appears to be coming from the node and not the Pod.

2. Ingress: This is the incoming traffic from the external world to Services. Ingress also allows and blocks particular communications with Services using rules for connections. Typically, there are two ingress solutions that function on different network stack regions: the service load balancer and the ingress controller.

Discovering Services

There are two ways Kubernetes discovers a Service:

1. Environment Variables: The kubelet service running on the node where your Pod runs is responsible for setting up environment variables for each active service in the format {SVCNAME}_SERVICE_HOST and {SVCNAME}_SERVICE_PORT. You must create the Service before the client Pods come into existence. Otherwise, those client Pods won't have their environment variables populated.

2. DNS: The DNS service is implemented as a Kubernetes service that maps to one or more DNS server Pods, which are scheduled just like any other Pod. Pods in the cluster are configured to use the DNS service, with a DNS search list that includes the Pod's own namespace and the cluster's default domain. A cluster-aware DNS server, such as CoreDNS, watches the Kubernetes API for new Services and creates a set of DNS records for each one. If DNS is enabled throughout your cluster, all Pods can automatically resolve Services by their DNS name. The Kubernetes DNS server is the only way to access ExternalName Services.

ServiceTypes for publishing Services:

Kubernetes Services provide you with a way of accessing a group of Pods, usually defined by using a label selector. This could be applications trying to access other applications within the cluster, or it could allow you to expose an application running in the cluster to the external world. Kubernetes ServiceTypes enable you to specify what kind of Service you want.

(Ahmet Alp Balkan, CC BY-SA 4.0)

The different ServiceTypes are:

1. ClusterIP: This is the default ServiceType. It makes the Service only reachable from within the cluster and allows applications within the cluster to communicate with each other. There is no external access.

2. LoadBalancer: This ServiceType exposes the Services externally using the cloud provider's load balancer. Traffic from the external load balancer is directed to the backend Pods. The cloud provider decides how it is load-balanced.

3. NodePort: This allows the external traffic to access the Service by opening a specific port on all the nodes. Any traffic sent to this Port is then forwarded to the Service.

4. ExternalName: This type of Service maps a Service to a DNS name by using the contents of the externalName field by returning a CNAME record with its value. No proxying of any kind is set up.

In a standard Kubernetes deployment, there are several networking variations you should be aware of. Below are the most common networking situations to know.

Container-to-container Networking

Pod-to-Pod Networking

Pod-to-pod networking can occur for pods within the same node or across nodes. Each of your nodes has a classless inter-domain routing (CIDR) block. This block is a defined set of unique IP addresses that are assigned to pods within that node. This ensures that each pod is provided with a unique IP regardless of which node it is in.

There are 2 types of communication.

• Inter-node communication

• Intra-node communication

Pod-to-Service Networking

Kubernetes is designed to allow pods to be replaced dynamically, as needed. This means that pod IP addresses are not durable unless special precautions are taken, such as for stateful applications. To address this issue and ensure that communication with and between pods is maintained, Kubernetes uses services.

Kubernetes services manage pod states and enable you to track pod IP addresses over time. These services abstract pod addresses by assigning a single virtual IP (a cluster IP) to a group of pod IPs. Then, any traffic sent to the virtual IP is distributed to the associated pods.

This service IP enables pods to be created and destroyed as needed without affecting overall communications. It also enables Kubernetes services to act as in-cluster load balancers, distributing traffic as needed among associated pods.

Internet-to-Service Networking

The final networking situation that is needed for most deployments is between the Internet and services. Whether you are using Kubernetes for internal or external applications, you generally need Internet connectivity. This connectivity enables users to access your services and distributed teams to collaborate.

When setting up external access, there are two techniques you need to use — egress and ingress. These are policies that you can set up with either whitelisting or blacklisting to control traffic into and out of your network.

Kubernetes Service provides the IP Address, a single DNS name, and a Load Balancer to a set of Pods. A Service identifies its member Pods with a selector. For a Pod to be a member of the Service, the Pod must have all of the labels specified in the selector. A label is an arbitrary key/value pair that is attached to an object. K8s Services are also a REST object and also an abstraction that defines a logical set of pods and a policy for accessing the pod set.

Services select Pods based on their labels. When a network request is made to the service, it selects all Pods in the cluster matching the service’s selector, chooses one of them, and forwards the network request to it. Let us look at the core attributes of any kind of service in Kubernetes:

• Label selector that locates pods

• ClusterIP IP address & assigned port number

• Port definitions

• Optional mapping of incoming ports to a targetPort

ClusterIP

ClusterIP is the default Service type in Kubernetes. In this Service, Kubernetes creates a stable IP Address that is accessible from all the nodes in the cluster. The scope of this service is confined within the cluster only. The main use case of this service is to connect our Frontend Pods to our Backend Pods as we don’t expose backend Pods to the outside world because of security reasons.

NodePort

NodePort exposes the Service on each Node’s IP at a static port (the NodePort). NodePort builds on top of ClusterIP to create a mapping from each Worker Node’s static IP on a specified (or Kubernetes has chosen) Port. We can contact the NodePort service, from outside the cluster, by requesting <Node IP>:<Nodeport>. The only main use case of NodePort Service is to expose our Pods to the outside world. Note: We can only expose the ports 30000-32767.

LoadBalancer

The LoadBalancer Service is a standard way for exposing our Nodes to the outside world or the internet. We have multiple Pods deployed on multiple Nodes, to access our application we can use any of the Public IP of any node and node port. But there are some problems in this scenario, like which Nodes IP we will provide to the clients and how will the traffic balance between the multiple nodes into the cluster. A simple solution for this will be LoadBalancer.

Ingress Controller

Ingress Controller is an intelligent Load Balancer. Ingress is a high-level abstraction responsible for allowing simple host or URL based HTTP routing. It is always implemented using a third-party proxy. These implementations are nothing but Ingress Controller. It is a Layer-7 load balancer.

The Domain Name System (DNS) is the networking system in place that allows us to resolve human-friendly names to unique IP addresses. By default, most Kubernetes clusters automatically configure an internal DNS service to provide a lightweight mechanism for service discovery. Kube-DNS and CoreDNS are two established DNS solutions for defining DNS naming rules and resolving pod and service DNS to their corresponding cluster IPs. With DNS, Kubernetes services can be referenced by name that will correspond to any number of backend pods managed by the service.

A DNS Pod consists of three separate containers:

• Kubedns: watches the Kubernetes master for changes in Services and Endpoints, and maintains in-memory lookup structures to serve DNS requests.

• Dnsmasq: adds DNS caching to improve performance.

• Sidecar: provides a single health check endpoint to perform health checks for dnsmasq and kubedns.

Lab 1 : Create ingress route to Nignx deployment

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress
spec:
  rules:
  - host: "nginx-ingress.kubernetes.saasx.io"
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: my-nginx-svc
            port:
              number: 80

Lab 2 : Create ingress route to Wordpress deployment

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress
spec:
  rules:
  - host: "wp-ingress.kubernetes.saasx.io"
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: wordpress
            port:
              number: 80

Recommended Application HW Sizing

We’ll cover the following topics, including hands-on practice on the functionalities of these concepts:

• What are PersistentVolumes and PersistentVolumeClaims?

• How and by whom are PersistentVolumes defined?

• How do PersistentVolumes differ from Volumes?

• How are PersistentVolumes and PersistentVolumeClaims created?

• How and by whom is persistent storage requested using PersistentVolumeClaims?

• Relationship between PV , PVC and Storage Volume

• Static provisioning

• Dynamic Provisioning

• Kubernetes PersistentVolumes (PVs) and PersistentVolumeClaims (PVCs)

  Storage management is essential in Kubernetes, especially in large environments where many users deploy multiple Pods. The users in this environment often need to configure storage for each Pod, and when making a change to existing applications, it must be made on all Pods, one after the other. To mitigate this time-consuming scenario and separate the details of how storage is provisioned from how it is consumed, we use PersistentVolumes (PVs) and PersistentVolumeClaims (PVCs).

  A PersistentVolume (PV) in Kubernetes is a pool of pre-provisioned storage resources in a Kubernetes cluster, that can be used across different user environments. Its lifecycle is separate from a Pod that uses the PersistentVolume.

  A PersistentVolumeClaim (PVC), is a process of storage requests from PVs by the users in Kubernetes. Kubernetes binds PVs with the PVCs based on the request and property set on those PVs. Kubernetes searches for PVs that correspond to the PVCs’ requested capacity and specified properties, so that each PVC can bind to a single PV.

  When there are multiple matches, you can use labels and selectors to bind a PVC to the right or a particular PV. This helps guard against a situation where a small PVC binds to a larger PV, since PVs and PVCs have a one-to-one relationship. When this happens, the remaining storage in the bound PVs are inaccessible to other users.

  NOTE: Both the PVCs and the Pod using them must be in the same namespace.

  The Difference Between PVs and PVCs in Kubernetes

  PVs and PVCs differ in provisioning, functionalities, and the person responsible for creating them, specifically :

  • PVs are created by the cluster administrator or dynamically by Kubernetes, whereas users/developers create PVCs.

  • PVs are cluster resources provisioned by an administrator, whereas PVCs are a user’s request for storage and resources.

  • PVCs consume PVs resources, but not vice versa.

  • A PV is similar to a node in terms of cluster resources, while a PVC is like a Pod in the context of cluster resource consumption.

  Difference between Volumes and PersistentVolumes

  Volumes and PersistentVolumes differ in the following ways:

  • A Volume separates storage from a container but binds it to a Pod, while PVs separate storage from a Pod.

  • The lifecycle of a Volume is dependent on the Pod using it, while the lifecycle of a PV is not.

  • A Volume enables safe container restart and allows sharing of data across containers, whereas a PV enables safe Pod termination or restart.

  • A separate manifest YAML file is needed to create a PV, but not required for a volume.

  PersistentVolumes and PersistentVolumeClaims Lifecycle

  The communication between PVs and PVCs consists of the following stages:

  • Provisioning: This is the process of creating a storage volume, which can be done statically or dynamically.

    • Static: The static way of provisioning a storage volume is when PVs are created before PVCs by an Administrator and exist in the Kubernetes API, waiting to be claimed by a user’s storage request, using PVC.

    • Dynamic: The dynamic way of storage provisioning involves creating PVs automatically, using StorageClass instead of manual provisioning of PersistentVolumes. Kubernetes will dynamically provision a volume specifically for this storage request, with the help of a StorageClass. (More about StorageClass and how it is used to provision PVs dynamically in the next part of this series).

• Binding: When an Administrator has provisioned PVs and users make a specific amount of storage requests (PVCs), a control loop in the master finds a matching PV and binds them together. If a matching volume does not exist, the claim will remain unbound until a volume match is created.

• Using: Pods use claims as volumes. The Kubernetes API checks the claim to find a bound PV and mounts it in the Pod for the users. When a claim is already bound to a PV, the bind remains unchanged as long as the user wants it. So user A’s bound PV can not be taken over by user B, if it is still in use by user A.

• Reclaiming: When a user is done with the volume usage, the resources must be freed up by deleting the PVC objects from Kubernetes. This practice will free the PV from the PVC, allowing reclamation of the resources, making it available for future use. What happens to the volumes afterwards is determined by the PersistentVolumeReclaimPolicy (PVRP) value specified in the configuration file. The PVRP provides three options of what you can do to a PersistentVolume after the claim has been deleted: retain, delete, or recycle.

  • Retain: This is the default reclaim policy. It is a process that allows manual deletion of the PV by the Administrator. When a PVC is deleted, the PV remains intact, including its data, thus making it unavailable for use by another claim. However, the Administrator can still manually reclaim the PersistentVolume.

  • Delete: When the reclaim policy is set to delete, the PV deletes automatically when the PVC is deleted and makes the space available. It is important to note that the dynamically provisioned PVs inherit the reclaim policy of their StorageClass, which defaults to Delete.

  • Recycle: This type of reclaim policy will scrub the data in the PV and make it available for another claim.

How to Create a PersistentVolume

Follow the below steps to create a PersistentVolume:

Step 1: Create the YAML file.

$ vim pv-config.yaml

Step 2: Copy and paste the below configuration file into the YAML manifest file created above.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-volume
spec:
  capacity:
    storage: 3Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/app/data"

The configuration above shows properties with different functionalities. In addition to the properties from the previous Kubernetes objects exercises, let’s look at accessModes, capacity, and storage properties:

• accessModes: This property defines how a PV should be mounted on the host. The value can be ReadWriteOnce, ReadOnlyMany, or ReadWriteMany. Below is a basic description of these values:

  • ReadWriteOnce: You can mount the PV as read-write by a single node.

  • ReadOnlyMany: You can mount the PV as read-only by multiple nodes.

  • ReadWriteMany: You can mount the PV as read-write by multiple nodes.

• capacity.storage: This property is used to set the amount of storage needed for the volume.

Step 3: Create the Persistent Volume using kubectl create command.

$ kubectl create -f pv-config.yaml
persistentvolume/my-volume created

Step 4: Check the created PV to see if it is available.

$ kubectl get pv

NAME       CAPACITY   ACCESS MODES   RECLAIM POLICY  STATUS     CLAIM    STORAGECLASS      REASON      AGE 
my-volume    3Gi       RWO              Retain       Available                         ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ 60s

Step 5: Check the description of the PersistentVolume by running kubectl describe command.

$ kubectl describe pv my-volume

Labels:          <none>
Annotations:     <none>
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:
Status:          Available
Claim:
Reclaim Policy:  Retain
Access Modes:    RWO
VolumeMode:      Filesystem
Capacity:        3Gi
Node Affinity:   <none>
Message:
Source:
    Type:          HostPath (bare host directory volume)
    Path:          /app/data
    HostPathType:
Events:            <none>

As seen above, the PersistentVolume is available and ready to serve a PVC storage request. The status will change from available to bound when it has been claimed.

NOTE: It is not advisable to use the hostpath volume type in a production environment.

How to Create a PersistentVolumeClaim in Kubernetes

Now that the PersistentVolume has been successfully created, the next step is to create a PVC that will claim the volume. Creating a PersistentVolumeClaim is similar to the method used to create the PersistentVolume above, with a few differences in terms of its properties and values. The value of the kind property will be PersistentVolumeClaim. The resources.request.storage field will also be added, and the value will be the provisioned PV capacity value (so 3Gi in our case).

The configuration will look like the manifest file below:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-claim
spec:
  accessModes:
    - ReadWriteOnce
   resources:
      requests:
        storage: 3Gi

Step 1: Create a YAML file.

$ vim pvc-config.yaml

Step 2: Copy and paste the above configuration file into the YAML file created above.

Step 3: Create the PVC by running kubectl create command.

$ kubectl create -f pvc-config.yaml
persistentvolumeclaim/my-claim created

Step 4: Check the status of the PVC by running kubectl get command.

$ kubectl get pvc my-claim

NAME       STATUS     VOLUME      CAPACITY   ACCESS MODES   STORAGECLASS   AGE
my-claim   Bound    my-volume       3Gi           RWO                      7m

Step 5: Check the description of the PVC.

$ kubectl describe pvc my-claim

Name:          my-claim
Namespace:     default
StorageClass: 
Status:        Bound
Volume:        my-volume
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      3Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Mounted By:    <none>
Events:        <none>

As shown above, the status indicates a “Bound” state, which means the PVC and PV are now bound together. Now, check the status of the PV once again with kubectl get command.

$ kubectl get pv my-volume

NAME       CAPACITY   ACCESS MODES   RECLAIM POLICY  STATUS     CLAIM        ⠀⠀⠀STORAGECLASS    REASON     AGE 
my-volume    3Gi       RWO              Retain       Bound   default/my-claim                          ⠀⠀⠀ 11m

The output shows that the status has changed from “available” when it is not yet bound to “Bound” because the PV has been claimed by the created PVC.

Step 6: Delete the PVC using kubectl delete command.

$ kubectl delete pvc my-claim

persistentvolumeclaim "my-claim" deleted

Step 7: Check both the PVC and PV status with kubectl get command.

$ kubectl get pvc my-claim

No resources found in default namespace.

$ kubectl get pv my-volume

NAME       CAPACITY   ACCESS MODES   RECLAIM POLICY  STATUS       CLAIM        ⠀⠀⠀STORAGECLASS    REASON     AGE 
my-volume    3Gi       RWO              Retain       Released   default/my-claim                          ⠀⠀ 21m

The output shows that the PV status has now changed from a “Bound” to a “Released” state, after the PVC is deleted.

PersistentVolumes States

A PV has different states, can be in any of these, and each has its own meaning, described as follows:

• Available: The PV is free and not yet claimed or bounded.

• Bound: The PV has been claimed or bounded.

• Released: The bounded PVC has been deleted, and the PV is free from its previous bounded state.

• Failed: The PV has failed its automatic reclamation.

PersistentVolumeClaims States

Each PVC, like the PV, has its own states that represent its current status.

• Bound: The PVC is bound to a PV.

• Pending: The PVC can not bind to a PV. This could be due to a higher storage request than the PV capacity, or the PV accessMode value is different from that of the PVC, etc.

How to Use PersistentVolumeClaim in a Pod

A Pod can access storage with the help of a PVC, which will be used as a volume. PVC can be used in a Pod by first declaring a “volumes” property in the Pod manifest file and specifying the claim name under the declared volume type “persistentVolumeClaim” property. It is essential that both the PVC and the Pod using it exist in the same namespace. This will allow the cluster to find the claim in the Pod’s namespace and use it to access the PersistentVolume that is bound to the PVC. Once created, the applications in the containers can read and write into the storage.The complete configuration file will look like this:

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: stclass-test
    image: nginx
    volumeMounts:
    - mountPath: "/app/data"
      name: my-volume
  volumes:
  - name: my-volume
    persistentVolumeClaim:
      claimName: my-claim 

The below steps will guide you through how to use a claim in a Pod. The PV and PVC have to be provisioned before creating the Pod. The claim name inside the Pod must also match the claim name in the running PVC.

Step 1: Create a YAML file.

$ vim pvc-pod.yaml

Step 2: Copy and paste the above Pod manifest file into the YAML file created above and create the Pod with kubectl create command.

$ kubectl create -f pvc-pod.yaml

pod/my-pod created

Step 3: Check the status and the description of the Pod.

$ kubectl get pod my-pod

NAME      READY   STATUS    RESTARTS    AGE
my-pod     1/1    Running      0       6m50s

$ kubectl describe pod my-pod

Name:         my-pod
Namespace:    default
Priority:     0
Node:         node01/172.17.0.57
Start Time:   Tue, 01 Dec 2020 11:44:08 +0000
Labels:       <none>
Annotations:  <none>
Status:       Running
IP:           10.244.1.2
IPs:
  IP:  10.244.1.2
Volumes:
  my-volume:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  my-claim
    ReadOnly:   false
  default-token-nlmxj:
    Type:       Secret (a volume populated by a Secret)
    SecretName: default-token-nlmxj
    Optional:   false
QoS Class:      BestEffort

Step 4: Exec into the Pod to test the PVC use case.

$ kubectl exec -it my-pod -- bin/bash

root@my-pod:/# 

Step 5: Use df -h together with the path to confirm the mount point.

root@my-pod:/# df -h /app/data

Filesystem                   		Size  	 Used 	    Avail       Use% Mountedon 
/dev/mapper/host01--vg-root  	        191G      22G        159G        13% /app/data

Step 6: Change into the mount directory and create a file using echo command.

root@my-pod:/# cd /app/data
root@my-pod:/app/data# echo "I love Kubermatic" > file.txt

Step 7: Check the created file and data using ls and cat commands, then exit the Pod once this has been confirmed.

root@my-pod:/app/data# ls
file.txt
root@my-pod:/app/data# cat file.txt
"I love Kubermatic"
root@my-pod:/app/data# exit

Step 8: Delete and recreate the Pod.

$ kubectl delete pod my-pod
pod "my-pod" deleted
$ kubectl get pods

No resources found in default namespace.

Recreate the Pod and check the status:

$ kubectl create -f pvc-pod.yaml
pod/my-pod created

$ kubectl get pod my-pod

NAME     READY     STATUS    RESTARTS   AGE
my-pod    1/1      Running     0         5s

Step 9: Exec into the Pod and check the previous file and data created if they exist in the new Pod.

$ kubectl exec -it my-pod -- bin/bash
root@my-pod:/# df -h /app/data

Filesystem                     Size    Used    Avail    Use%   Mountedon 
/dev/mapper/host01--vg-root    191G     22G    159G     13%    /app/data

root@my-pod:/# cd /app/data
root@my-pod:/app/data# ls
file.txt
root@my-pod:/app/data# cat file.txt
"I love Kubermatic"

You can see that the file and data created in the deleted Pod above are still there and were taken up by the new Pod.

Summary: PersistentVolumes and PersistentVolumeClaims are Kubernetes objects that work in tandem to give your Kubernetes applications a higher level of persistence, either statically or dynamically, with the help of StorageClass.

This comprehensive guide on Kubernetes architecture aims to explain each kubernetes component in detail with illustrations.

So, If you want to,

1. Understand kubernetes architecture

2. Understand the workflow of Kubernetes core components

Then you’ll love this guide.

TABLE OF CONTENTS

Kubernetes Architecture

The following Kubernetes architecture diagram shows all the components of the Kubernetes cluster and how external systems connect to the Kubernetes cluster.

The first and foremost thing you should understand about Kubernetes is, it is a distributed system. Meaning, it has multiple components spread across different servers over a network. These servers could be Virtual machines or bare metal servers. We call it a Kubernetes cluster.

A Kubernetes cluster consists of control plane nodes and worker nodes.

Control Plane

The control plane is responsible for container orchestration and maintaining the desired state of the cluster. It has the following components.

1. kube-apiserver

2. etcd

3. kube-scheduler

4. kube-controller-manager

5. cloud-controller-manager

Worker Node

The Worker nodes are responsible for running containerized applications. The worker Node has the following components.

1. kubelet

2. kube-proxy

3. Container runtime

Kubernetes Control Plane Components

First, let’s take a look at each control plane component and the important concepts behind each component.

1. kube-apiserver

The kube-api server is the central hub of the Kubernetes cluster that exposes the Kubernetes API.

End users, and other cluster components, talk to the cluster via the API server. Very rarely monitoring systems and third-party services may talk to API servers to interact with the cluster.

The communication between the API server and other components in the cluster happens over TLS to prevent unauthorized access to the cluster.

Kubernetes api-server is responsible for the following

1. API management: Exposes the cluster API endpoint and handles all API requests.

2. Authentication (Using client certificates, bearer tokens, and HTTP Basic Authentication) and Authorization (ABAC and RBAC evaluation)

3. Processing API requests and validating data for the API objects like pods, services, etc. (Validation and Mutation Admission controllers)

4. It is the only component that communicates with etcd.

5. api-server coordinates all the processes between the control plane and worker node components.

6. api-server has a built-in bastion apiserver proxy. It is part of the API server process. It is primarily used to enable access to ClusterIP services from outside the cluster, even though these services are typically only reachable within the cluster itself.

2. etcd

Kubernetes is a distributed system and it needs an efficient distributed database like etcd that supports its distributed nature. It acts as both a backend service discovery and a database. You can call it the brain of the Kubernetes cluster.

2. Distributed: etcd is designed to run on multiple nodes as a cluster without sacrificing consistency.

So how etcd works with Kubernetes?

To put it simply, when you use kubectl to get kubernetes object details, you are getting it from etcd. Also, when you deploy an object like a pod, an entry gets created in etcd.

In a nutshell, here is what you need to know about etcd.

2. etcd allows a client to subscribe to events using Watch() API . Kubernetes api-server uses the etcd’s watch functionality to track the change in the state of an object.

4. etcd stores all objects under the /registry directory key in key-value format. For example, information on a pod named Nginx in the default namespace can be found under /registry/pods/default/nginx

Also, etcd it is the only Statefulset component in the control plane.

3. kube-scheduler

The kube-scheduler is responsible for scheduling pods on worker nodes.

When you deploy a pod, you specify the pod requirements such as CPU, memory, affinity, taints or tolerations, priority, persistent volumes (PV), etc. The scheduler’s primary task is to identify the create request and choose the best node for a pod that satisfies the requirements.

The following image shows a high-level overview of how the scheduler works.

In a Kubernetes cluster, there will be more than one worker node. So how does the scheduler select the node out of all worker nodes?

Here is how the scheduler works.

1. To choose the best node, the Kube-scheduler uses filtering and scoring operations.

2. In filtering, the scheduler finds the best-suited nodes where the pod can be scheduled. For example, if there are five worker nodes with resource availability to run the pod, it selects all five nodes. If there are no nodes, then the pod is unschedulable and moved to the scheduling queue. If It is a large cluster, let’s say 100 worker nodes, and the scheduler doesn’t iterate over all the nodes. There is a scheduler configuration parameter called percentageOfNodesToScore. The default value is typically 50%. So it tries to iterate over 50% of nodes in a round-robin fashion. If the worker nodes are spread across multiple zones, then the scheduler iterates over nodes in different zones. For very large clusters the default percentageOfNodesToScore is 5%.

4. Once the node is selected, the scheduler creates a binding event in the API server. Meaning an event to bind a pod and node.

Things should know about a scheduler.

1. It is a controller that listens to pod creation events in the API server.

2. The scheduler has two phases. Scheduling cycle and the Binding cycle. Together it is called the scheduling context. The scheduling cycle selects a worker node and the binding cycle applies that change to the cluster.

4. You can create custom schedulers and run multiple schedulers in a cluster along with the native scheduler. When you deploy a pod you can specify the custom scheduler in the pod manifest. So the scheduling decisions will be taken based on the custom scheduler logic.

4. Kube Controller Manager

As per the official documentation,

In Kubernetes, controllers are control loops that watch the state of your cluster, then make or request changes where needed. Each controller tries to move the current cluster state closer to the desired state.

Let’s say you want to create a deployment, you specify the desired state in the manifest YAML file (declarative approach). For example, 2 replicas, one volume mount, configmap, etc. The in-built deployment controller ensures that the deployment is in the desired state all the time. If a user updates the deployment with 5 replicas, the deployment controller recognizes it and ensures the desired state is 5 replicas.

Kube controller manager is a component that manages all the Kubernetes controllers. Kubernetes resources/objects like pods, namespaces, jobs, replicaset are managed by respective controllers. Also, the kube scheduler is also a controller managed by Kube controller manager.

Following is the list of important built-in Kubernetes controllers.

1. Deployment controller

2. Replicaset controller

3. DaemonSet controller

5. CronJob Controller

6. endpoints controller

7. namespace controller

9. Node controller

Here is what you should know about the Kube controller manager.

1. It manages all the controllers and the controllers try to keep the cluster in the desired state.

2. You can extend kubernetes with custom controllers associated with a custom resource definition.

5. Cloud Controller Manager (CCM)

When kubernetes is deployed in cloud environments, the cloud controller manager acts as a bridge between Cloud Platform APIs and the Kubernetes cluster.

This way the core kubernetes core components can work independently and allow the cloud providers to integrate with kubernetes using plugins. (For example, an interface between kubernetes cluster and AWS cloud API)

Cloud controller integration allows Kubernetes cluster to provision cloud resources like instances (for nodes), Load Balancers (for services), and Storage Volumes (for persistent volumes).

Cloud Controller Manager contains a set of cloud platform-specific controllers that ensure the desired state of cloud-specific components (nodes, Loadbalancers, storage, etc). Following are the three main controllers that are part of the cloud controller manager.

1. Node controller: This controller updates node-related information by talking to the cloud provider API. For example, node labeling & annotation, getting hostname, CPU & memory availability, nodes health, etc.

2. Route controller: It is responsible for configuring networking routes on a cloud platform. So that pods in different nodes can talk to each other.

3. Service controller: It takes care of deploying load balancers for kubernetes services, assigning IP addresses, etc.

Following are some of the classic examples of cloud controller manager.

1. Deploying Kubernetes Service of type Load balancer. Here Kubernetes provisions a Cloud-specific Loadbalancer and integrates with Kubernetes Service.

2. Provisioning storage volumes (PV) for pods backed by cloud storage solutions.

Overall Cloud Controller Manager manages the lifecycle of cloud-specific resources used by kubernetes.

Kubernetes Worker Node Components

Now let’s look at each of the worker node components.

1. Kubelet

Kubelet is an agent component that runs on every node in the cluster. t does not run as a container instead runs as a daemon, managed by systemd.

It is responsible for registering worker nodes with the API server and working with the podSpec (Pod specification – YAML or JSON) primarily from the API server. podSpec defines the containers that should run inside the pod, their resources (e.g. CPU and memory limits), and other settings such as environment variables, volumes, and labels.

It then brings the podSpec to the desired state by creating containers.

To put it simply, kubelet is responsible for the following.

1. Creating, modifying, and deleting containers for the pod.

2. Responsible for handling liveliness, readiness, and startup probes.

3. Responsible for Mounting volumes by reading pod configuration and creating respective directories on the host for the volume mount.

4. Collecting and reporting Node and pod status via calls to the API server.

Kubelet is also a controller where it watches for pod changes and utilizes the node’s container runtime to pull images, run containers, etc.

Other than PodSpecs from the API server, kubelet can accept podSpec from a file, HTTP endpoint, and HTTP server. A good example of “podSpec from a file” is Kubernetes static pods.

Static pods are controlled by kubelet, not the API servers.

This means you can create pods by providing a pod YAML location to the Kubelet component. However, static pods created by Kubelet are not managed by the API server.

Here is a real-world example use case of the static pod.

While bootstrapping the control plane, kubelet starts the api-server, scheduler, and controller manager as static pods from podSpecs located at /etc/kubernetes/manifests

Following are some of the key things about kubelet.

1. Kubelet uses the CRI (container runtime interface) gRPC interface to talk to the container runtime.

2. It also exposes an HTTP endpoint to stream logs and provides exec sessions for clients.

3. Uses the CSI (container storage interface) gRPC to configure block volumes.

4. It uses the CNI plugin configured in the cluster to allocate the pod IP address and set up any necessary network routes and firewall rules for the pod.

2. Kube proxy

To understand kube proxy, you need to have a basic knowledge of Kubernetes Service & endpoint objects.

Service in Kubernetes is a way to expose a set of pods internally or to external traffic. When you create the service object, it gets a virtual IP assigned to it. It is called clusterIP. It is only accessible within the Kubernetes cluster.

The Endpoint object contains all the IP addresses and ports of pod groups under a Service object. The endpoints controller is responsible for maintaining a list of pod IP addresses (endpoints). The service controller is responsible for configuring endpoints to a service.

You cannot ping the ClusterIP because it is only used for service discovery, unlike pod IPs which are pingable.

Now let’s understand Kube Proxy.

When you expose pods using a Service (ClusterIP), Kube-proxy creates network rules to send traffic to the backend pods (endpoints) grouped under the Service object. Meaning, all the load balancing, and service discovery are handled by the Kube proxy.

So how does Kube-proxy work?

Kube proxy talks to the API server to get the details about the Service (ClusterIP) and respective pod IPs & ports (endpoints). It also monitors for changes in service and endpoints.

Kube-proxy then uses any one of the following modes to create/update rules for routing traffic to pods behind a Service

2. 1. rr: round-robin : It is the default mode.

   2. lc: least connection (smallest number of open connections)

   3. dh: destination hashing

   4. sh: source hashing

   5. sed: shortest expected delay

   6. nq: never queue

3. Userspace (legacy & not recommended)

4. Kernelspace: This mode is only for windows systems.

3. Container Runtime

Container runtime runs on all the nodes in the Kubernetes cluster. It is responsible for pulling images from container registries, running containers, allocating and isolating resources for containers, and managing the entire lifecycle of a container on a host.

To understand this better, let’s take a look at two key concepts:

1. Container Runtime Interface (CRI): It is a set of APIs that allows Kubernetes to interact with different container runtimes. It allows different container runtimes to be used interchangeably with Kubernetes. The CRI defines the API for creating, starting, stopping, and deleting containers, as well as for managing images and container networks.

2. Open Container Initiative (OCI): It is a set of standards for container formats and runtimes

So how does Kubernetes make use of the container runtime?

As we learned in the Kubelet section, the kubelet agent is responsible for interacting with the container runtime using CRI APIs to manage the lifecycle of a container. It also gets all the container information from the container runtime and provides it to the control plane.

1. When there is a new request for a pod from the API server, the kubelet talks to CRI-O daemon to launch the required containers via Kubernetes Container Runtime Interface.

2. CRI-O checks and pulls the required container image from the configured container registry using containers/image library.

3. CRI-O then generates OCI runtime specification (JSON) for a container.

4. CRI-O then launches an OCI-compatible runtime (runc) to start the container process as per the runtime specification.

Kubernetes Cluster Addon Components

Apart from the core components, the kubernetes cluster needs addon components to be fully operational. Choosing an addon depends on the project requirements and use cases.

Following are some of the popular addon components that you might need on a cluster.

1. CNI Plugin (Container Network Interface)

2. CoreDNS (For DNS server): CoreDNS acts as a DNS server within the Kubernetes cluster. By enabling this addon, you can enable DNS-based service discovery.

3. Metrics Server (For Resource Metrics): This addon helps you collect performance data and resource usage of Nodes and pods in the cluster.

4. Web UI (Kubernetes Dashboard): This addon enables the Kubernetes dashboard for managing the object via web UI.

1. CNI Plugin

It is not specific to Kubernetes. With CNI container networking can be standardized across container orchestration tools like Kubernetes, Mesos, CloudFoundry, Podman, Docker, etc.

When it comes to container networking, companies might have different requirements such as network isolation, security, encryption, etc. As container technology advanced, many network providers created CNI-based solutions for containers with a wide range of networking capabilities. You can call it as CNI-Plugins

This allows users to choose a networking solution that best fits their needs from different providers.

How does CNI Plugin work with Kubernetes?

1. The Kube-controller-manager is responsible for assigning pod CIDR to each node. Each pod gets a unique IP address from the pod CIDR.

2. Kubelet interacts with container runtime to launch the scheduled pod. The CRI plugin which is part of the Container runtime interacts with the CNI plugin to configure the pod network.

3. CNI Plugin enables networking between pods spread across the same or different nodes using an overlay network.

Following are high-level functionalities provided by CNI plugins.

1. Pod Networking

2. Pod network security & isolation using Network Policies to control the traffic flow between pods and between namespaces.

Some popular CNI plugins include:

1. Calico

2. Flannel

3. Weave Net

4. Cilium (Uses eBPF)

5. Amazon VPC CNI (For AWS VPC)

6. Azure CNI (For Azure Virtual network)Kubernetes networking is a big topic and it differs based on the hosting platforms.

Kubernetes Configmaps

If we have many object files, it won’t be easy to manage the environment data stored within the query files. We can take this information out of the pod definition file and execute it centrally using Configuration Maps. ConfigMaps are used to provide configuration data in key-value pairs in K8s. There are two phases involved in configuring ConfigMaps. The first is to create the ConfigMaps, and the second to inject them into the pod.

ConfigMaps

There are two ways of creating a config map:

i) The imperative way – without using a ConfigMap definition file ii) Declarative way by using a Configmap definition file.

Create a ConfigMap

ConfigMap can be written manually as a YAML representation and load it into Kubernetes, or you can also apply the kubectl create configmap command to create it from the CLI.

Create a ConfigMap Using kubectl

Run the kubectl create configmap cmd to create ConfigMaps from directories, files, or literal values:

kubectl create configmap <map-name> <data-source>

Where <map-name> is the name you want for ConfigMap and <data-source> is the directory, file, or literal value to draw the data from.

ConfigMaps and Pods

You can write a Pod spec that attributes to a ConfigMap and configures the container based on the data in the ConfigMap. Note that the Pod and the ConfigMap must exist in the same namespace.

Let’s look at an instance now, ConfigMap, with some keys with single values and other keys where the value looks like a fragment of a configuration format.

apiVersion: v1
kind: ConfigMap
metadata:
  name: game-demo
data:
  # property-like keys; each key maps to a simple value
  player_initial_lives: "3"
  ui_properties_file_name: "user-interface.properties"

  # file-like keys
  game.properties: |
    enemy.types=aliens,monsters
    player.maximum-lives=5
  user-interface.properties: |
    color.good=purple
    color.bad=yellow
    allow.textmode=true

There are four various ways you can use a ConfigMap to configure a container in a Pod:

1. Entrypoint of the containers via command line.

2. Environment variables for a container.

3. Also, we can add a file in read-only volume for the application to read.

4. Manually written Code to run inside the Pod that uses the K8s API to read a ConfigMap.

Using ConfigMaps

ConfigMaps can be mounted as data volumes. Other parts of the system can also use ConfigMaps without being directly exposed to the Pod. For example, ConfigMaps can hold data that different system parts should use for configuration. The usual way of using ConfigMaps is to configure environments for containers running in a Pod in the same namespace. You can also use ConfigMap separately.

Immutable ConfigMaps

The K8s beta feature Immutable Secrets and ConfigMaps gives an option to set specific Secrets and ConfigMaps as immutable. And for clusters that extensively use ConfigMaps, restricting modifications to their data has the following advantages:

• Protects you from accidental updates that could cause applications outages

• Improves the performance of your cluster by significantly reducing the load on the Kube-API server by closing watches for ConfigMaps marked as immutable.

The Immutable Ephemeral Volumes control this feature gate. You can build an immutable ConfigMap by fixing the immutable field to true. For example:

apiVersion: v1
kind: ConfigMap
metadata:
  ...
data:
  ...
immutable: true

Kubernetes Secrets

Kubernetes Secrets

Secrets 🔐 are used to hold sensitive data like passwords or keys. They’re similar to configMap except that they’re stored in an encoded or hashed format with config maps.

There are two levels involved in working with secrets. First, to create the secret, and second, to introduce it into the pod. Putting confidential data in secret is safer and adaptable rather than putting it in a container image or a Pod definition.

To use a secret, a pod has to reference the secret. There are three ways to use a secret with a Pod:

• As a file in a volume mounted on one or more of its containers.

• As a container environment variable.

• kubelet uses secrets by pulling images for the Pod.

Built-in Secrets

Built-in Secrets is where Service accounts automatically create and Attach Secrets with API credentials. K8s automatically creates secrets that include credentials for reaching the API and automatically alters your pods to use this type of secret. The automatic and also the use of API credentials can be overridden or disabled if wanted.

Creating a Secret

There are several options to create a Secret:

• Create Secret using kubectl command

• We can create a Secret from the config file

• Create Secret using customise

Editing a Secret

Run the following command to edit a Secret:

 kubectl edit secrets mysecret

Using Secrets

Secrets can be attached as data volumes or exhibited as environment variables used by a container in a Pod. Other parts of the system can also use secrets without being directly exposed to the Pod.

Updating Secrets and ConfigMaps

Kubernetes manages our environment variables, which means we don’t have to change code or rebuild containers when changing the variable’s value. The updation is a two-step process because pods cache the value of environment variables during startup.

First, update the values:

kubectl create configmap language --from-literal=LANGUAGE=Spanish \
        -o yaml --dry-run | kubectl replace -f -
kubectl create secret generic apikey --from-literal=API_KEY=098765 \
        -o yaml --dry-run | kubectl replace -f -

Then, restart the pods. This can be done in various ways, such as forcing a new deployment. The quick and easy way is to delete the pods manually and have the deployment automatically spin up new ones.

kubectl delete pod -l name=envtest

It is most suitable to create your Secrets and ConfigMaps using the above method so kubectl can record its annotation for tracking changes to the resource in the spec. Another way of doing this is using the –save-config command-line option when running kubectl create secret|configmap.

What is Elastic Transcoder?

• Elastic Transcoder is an aws service used to convert the media files stored in an S3 bucket into the media files in different formats supported by different devices.

• Elastic Transcoder is a media transcoder in the cloud.

• It is used to convert media files from their original source format into different formats that will play on smartphones, tablets, PC's, etc.

• It provides transcoding presets for popular output formats means that you don't need to guess about which settings work best on particular devices.

• If you use Elastic Transcoder, then you need to pay based on the minutes that you transcode and the resolution at which you transcode.

Components of Elastic Transcoder

Elastic Transcoder consists of four components:

• Jobs

• Pipelines

• Presets

• Notifications

• Jobs The main task of the job is to complete the work of transcoding. Each job can convert a file up to 30 formats. For example, if you want to convert a media file into eight different formats, then a single job creates files in eight formats. When you create a job, you need to specify the name of the file that you want to transcode.

• Pipelines Pipelines are the queues that consist of your transcoding jobs. When you create a job, then you need to specify which pipeline you want to add your job. If you want a job to create more than one format, Elastic Transcoder creates the files for each format in the order you specify the formats in a job. You can create either of the two pipelines, i.e., standard-priority jobs and high-priority jobs. Mainly jobs go into the standard-priority jobs. Sometimes you want to transcode the file immediately; the high-priority pipeline is used.

• Presets Presets are the templates that contain the settings for transcoding the media file from one format to another format. Elastic transcoder consists of some default presets for common formats. You can also create your own presets that are not included in the default presets. You need to specify a preset that you want to use when you create a job.

• Notifications Notification is an optional field which you can configure with the Elastic Transcoder. Notification Service is a service that keeps you updated with the status of your job: when Elastic Transcoder starts processing your job, when Elastic Transcoder finishes its job, whether the Elastic Transcoder encounters an error condition or not.You can configure Notifications when you create a pipeline.

How A Cloud Uses Elastic Transcoder

Suppose I uploaded the mp4 file in S3 bucket. As soon as uploading is completed, it triggers a Lambda function. Lambda function will then invoke Elastic Transcoder. Elastic Transcoder converts the mp4 file into different formats so that the file can be opened in iphone, Laptop, etc. Once it has completed the transcoding, it stores the transcoded files in S3 bucket.

What is a Kubernetes Namespace?

• Apply access controls—namespaces let you define role-based access control (RBAC). You can define a role object type and assign it using role binding. The role you define is applied to a namespace, and RoleBinding is applied to specific objects within this namespace. Using this technique can help you improve the security of your cluster.

In a new cluster, Kubernetes automatically creates the following namespaces: default (for user workloads) and three namespaces for the Kubernetes control plane: kube-node-lease, kube-public, and kube-system. Kubernetes also allows admins to manually create custom namespaces.

In this article:

Kubernetes Namespaces Concepts

There are two types of Kubernetes namespaces: Kubernetes system namespaces and custom namespaces.

Default Kubernetes namespaces

Here are the four default namespaces Kubernetes creates automatically:

• default—a default space for objects that do not have a specified namespace.

• kube-system—a default space for Kubernetes system objects, such as kube-dns and kube-proxy, and add-ons providing cluster-level features, such as web UI dashboards, ingresses, and cluster-level logging.

• kube-public—a default space for resources available to all users without authentication.

• kube-node-lease—a default space for objects related to cluster scaling.

Custom Kubernetes namespaces

Admins can create as many Kubernetes namespaces as necessary to isolate workloads or resources and limit access to specific users. Here is how to create a namespace using kubectl:

kubectl create ns mynamespace

The Hierarchical Namespace Controller (HNC)

Hierarchical namespaces are an extension to the Kubernetes namespaces mechanism, which allows you to organize groups of namespaces that have a common owner. For example, when a cluster is shared by multiple teams, each team can have a group of namespaces that belong to them.

With hierarchical namespaces, you can create a team namespace, and under it namespaces for specific workloads. You don’t need cluster-level permission to create a namespace within your team namespace, and you also have the flexibility to apply different RBAC rules and network security groups at each level of the namespace hierarchy.

When Should You Use Multiple Kubernetes Namespaces?

In small projects or teams, where there is no need to isolate workloads or users from each other, it can be reasonable to use the default Kubernetes namespace. Consider using multiple namespaces for the following reasons:

• Isolation—if you have a large team, or several teams working on the same cluster, you can use namespaces to create separation between projects and microservices. Activity in one namespace never affects the other namespaces.

• Development stages—if you use the same Kubernetes cluster for multiple stages of the development lifecycle, it is a good idea to separate development, testing, and production environments. You do not want errors or instability in testing environments to affect production users. Ideally, you should use a separate cluster for each environment, but if this is not possible, namespaces can create this separation.

• Permissions—it might be necessary to define separate permissions for different resources in your cluster. You can define separate RBAC rules for each namespace, ensuring that only authorized roles can access the resources in the namespace. This is especially important for mission critical applications, and to protect sensitive data in production deployments.

• Resource control—you can define resource limits at the namespace level, ensuring each namespace has access to a certain amount of CPU and memory resources. This enables separating cluster resources among several projects and ensuring each project has the resources it needs, leaving sufficient resources for other projects.

• Performance—Kubernetes API provides better performance when you define namespaces in the cluster. This is because the API has less items to search through when you perform specific operations.

Quick Tutorial: Working with Kubernetes Namespaces

Let’s see how to perform basic namespace operations—creating a namespace, viewing existing namespaces, and creating a pod in a namespace.

Creating Namespaces

You can create a namespace with a simple kubectl command, like this:

kubectl create namespace mynamespace

This will create a namespace called mynamespace, with default configuration. If you want more control over the namespace, you can create a YAML file and apply it. Here is an example of a namespace YAML file:

kind: Namespace
apiVersion: v1
metadata:
  name: mynamespace
  labels:
    name: mynamespace
kubectl apply -f test.yaml

Viewing Namespaces

To list all the namespaces currently active in your cluster, run this command:

kubectl get namespace

The output will look something like this:

NAME             STATUS   AGE
default          Active   3d
kube-node-lease  Active   3d
kube-public      Active   3d
kube-system      Active   3d
mynamespace      Active   1d

Creating Resources in the Namespace

When you create a resource in Kubernetes without specifying a namespace, it is automatically created in the current namespace.

For example, the following pod specification does not specify a namespace:

apiVersion: v1
kind: Pod
metadata:
  name: mypod
  labels:
    name: mypod
spec:
  containers:
 —name: mypod
    image: nginx

When you apply this pod specification, the following will happen:

• If you did not create any namespaces in your cluster, the pod will be created in the default namespace

• If you created a namespace and are currently running in it, the pod will be created in that namespace.

How can you explicitly create a resource in a specific namespace?

There are two ways to do this:

1. Use the –namespace flag when creating the resource, like this:

kubectl apply -f pod.yaml --namespace=mynamespace

1. Specify a namespace in the YAML specification of the resource. Here is what it looks like in a pod specification:

apiVersion: v1
kind: Pod
metadata:
  name: mypod
  namespace: mynamespace
  labels:
    name: mypod
...

Important notes:

• Note that if your YAML specification specifies one namespace, but the apply command specifies another namespace, the command will fail.

• If you try to work with a Kubernetes resource in a different namespace, kubectl will not find the resource. Us the –namespace flag to work with resources in other namespaces, like this:

kubectl get pods --namespace=mynamespace

What Are Kubernetes Nodes?

Each node runs two main components—a kubelet and a container runtime. The kubelet is in charge of facilitating communication between the control plane and the node. The container runtime is in charge of pulling the relevant container image from a registry, unpacking containers, running them on the node, and communicating with the operating system kernel.

In this article:

Kubernetes Node Components

Here are three main Kubernetes node components:

kubelet

The Kubelet is responsible for managing the deployment of pods to Kubernetes nodes. It receives commands from the API server and instructs the container runtime to start or stop containers as needed.

kube-proxy

A network proxy running on each Kubernetes node. It is responsible for maintaining network rules on each node. Network rules enable network communication between nodes and pods. Kube-proxy can directly forward traffic or use the operating system packet filter layer.

Container runtime

The software layer responsible for running containers. There are several container runtimes supported by Kubernetes, including Containerd, CRI-O, Docker, and other Kubernetes Container Runtime Interface (CRI) implementations.

Working with Kubernetes Nodes: 4 Basic Operations

Here is how to perform common operations on a Kubernetes node.

1. Adding Node to a Cluster

You can manually add nodes to a Kubernetes cluster, or let the kubelet on that node self-register to the control plane. Once a node object is created manually or by the kubelet, the control plane validates the new node object.

Adding nodes automatically

The example below is a JSON manifest that creates a node object. Kubernetes then checks that the kubelet registered to the API server matches the node’s metadata.name field. Only healthy nodes running all necessary services are eligible to run the pod. If the check fails, the node is ignored until it becomes healthy.

{
  "kind": "Node",
  "apiVersion": "v1",
  "metadata": {
    "name": "<node-ip-address>",
    "labels": {
      "name": "<node-logical-name>"
    }
  }
}

Defining node capacity

Nodes that self-register with the API Server report their CPU and memory volume capacity after the node object is created. However, When manually creating the node, administrators need to set up capacity demands. Once this information is defined, the Kubernetes scheduler assigns resources to all pods running on a node. The scheduler is responsible for ensuring that requests do not exceed node capacity.

2. Modifying Node Objects

You can use kubectl to manually create or modify node objects, overriding the settings defined in --register-node. You can, for example:

• Use labels on nodes and node selectors to control scheduling. You can, for example, limit a pod to be eligible only for running on a subset of available nodes.

• Mark a node as unschedulable to prevent the scheduler from adding new pods to the node. This action does not affect pods running on the node. You can use this option in preparation for maintenance tasks like node reboot. To mark the node as unschedulable, you can run: kubectl cordon $NODENAME.

3. Checking Node Status

There are three primary commands you can use to determine the status of a node and the resources running on it.

kubectl describe nodes

Run the command kubectl describe nodes my-node to get node information including:

• HostName—reported by the node operating system kernel. You can report a different value for HostName using the kubelet flag --hostname-override.

• InternalIP—enables traffic to be routed to the node within the cluster.

• ExternalID—an IP that can be used to access the node from outside the cluster.

• Conditions—system resource issues including CPU and memory utilization. This section shows error conditions like OutOfDisk, MemoryPressure, and DiskPressure.

• Events—this section shows issues occurring in the environment, such as eviction of pods.

kubectl describe pods

You can use this command to get information about pods running on a node:

• Pod information—labels, resource requirements, and containers running in the pod

• Pod ready state—if a pod appears as READY, it means it passed the last readiness check.

• Container state—can be Waiting, Running, or Terminated.

• Restart count—how often a container has been restarted.

• Log events—showing activity on the pod, indicating which component logged the event, for which object, a Reason and a Message explaining what happened.

4. Understanding the Node Controller

The node controller is the control plane component responsible for managing several aspects of the node’s lifecycle. Here are the three main roles of the node controller:

Assigning CIDR addresses

When the node is registered, the node controller assigns a Cross Inter-Domain Routing (CIDR) block (if CIDR assignment is enabled).

Updating internal node lists

The node controller maintains an internal list of nodes. It needs to be updated constantly with the list of machines available by the cloud provider. This list enables the node controller to ensure capacity is met.

When a node is unhealthy, the node controller checks if the host machine for that node is available. If the VM is not available, the node controller deletes the node from the internal list. If Kubernetes is running on a public or private cloud, the node controller can send a request to create a new node, to maintain cluster capacity.

Monitoring the health of nodes

Here are several tasks the node controller is responsible for:

• Checking the state of all nodes periodically, with the period determined by the --node-monitor-period flag.

• Updating the NodeReady condition to ConditionUnknown if the node becomes unreachable and the node controller no longer receives heartbeats.

• Evicting all pods from the node. If the node remains unreachable, the node controller uses graceful termination to evict the pods. Timeouts are set by default to 40 seconds, before reporting ConditionUnknown. Five minutes later, the node controller starts evicting pods.

Kubernetes Node Security

kubelet

This component is the main node agent for managing individual containers that run in a pod. Vulnerabilities associated with the kubelet are constantly discovered, meaning that you need to regularly upgrade the kubelet versions and apply the latest patches. Access to the kubelet is not authenticated by default, so you should implement strong authentication measures to restrict access.

kube-proxy

This component handles request forwarding according to network rules. It is a network proxy that supports various protocols (i.e. TCP, UDP) and allows Kubernetes services to be exposed. There are two ways to secure kube-proxy:

• If proxy configuration is maintained via the kubeconfig file, restrict file permissions to ensure unauthorized parties cannot tamper with proxy settings.

• Ensure that communication with the API server is only done over a secured port, and always require authentication and authorization.

Hardened Node Security

You can harden your noder security by following these steps:

• Ensure the host is properly configured and secure—check your configuration to ensure it meets the CIS Benchmarks standards.

• Control access to sensitive ports—ensure the network blocks access to ports that kubelet uses. Limit Kubernetes API server access to trusted networks.

• Limit administrative access to nodes—ensure your Kubernetes nodes have restricted access. You can handle tasks like debugging and without having direct access to a node.

Isolation of Sensitive Workloads

You should run any sensitive workload on dedicated machines to minimize the impact of a breach. Isolating workloads prevents an attacker from accessing sensitive applications through lower-priority applications on the same host or with the same container runtime. Attackers can only exploit the kubelet credentials of compromised nodes to access secrets that are mounted on those nodes. You can use controls such as node pools, namespaces, tolerations and taints to isolate your workloads.

Taints and Tolerations

Taints and tolerations work together to ensure that pods are not scheduled onto inappropriate nodes. One or more taints are applied to a node; this marks that the node should not accept any pods that do not tolerate the taints.

Concepts

kubectl taint nodes node1 key1=value1:NoSchedule

places a taint on node node1. The taint has key key1, value value1, and taint effect NoSchedule. This means that no pod will be able to schedule onto node1 unless it has a matching toleration.

To remove the taint added by the command above, you can run:

kubectl taint nodes node1 key1=value1:NoSchedule-

You specify a toleration for a pod in the PodSpec. Both of the following tolerations "match" the taint created by the kubectl taint line above, and thus a pod with either toleration would be able to schedule onto node1:

tolerations:
- key: "key1"
  operator: "Equal"
  value: "value1"
  effect: "NoSchedule"

tolerations:
- key: "key1"
  operator: "Exists"
  effect: "NoSchedule"

Here's an example of a pod that uses tolerations:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    env: test
spec:
  containers:
  - name: nginx
    image: nginx
    imagePullPolicy: IfNotPresent
  tolerations:
  - key: "example-key"
    operator: "Exists"
    effect: "NoSchedule"

The default value for operator is Equal.

A toleration "matches" a taint if the keys are the same and the effects are the same, and:

• the operator is Exists (in which case no value should be specified), or

• the operator is Equal and the values are equal.

Note:

There are two special cases:

An empty key with operator Exists matches all keys, values and effects which means this will tolerate everything.

An empty effect matches all effects with key key1.

The above example used effect of NoSchedule. Alternatively, you can use effect of PreferNoSchedule. This is a "preference" or "soft" version of NoSchedule -- the system will try to avoid placing a pod that does not tolerate the taint on the node, but it is not required. The third kind of effect is NoExecute, described later.

You can put multiple taints on the same node and multiple tolerations on the same pod. The way Kubernetes processes multiple taints and tolerations is like a filter: start with all of a node's taints, then ignore the ones for which the pod has a matching toleration; the remaining un-ignored taints have the indicated effects on the pod. In particular,

• if there is at least one un-ignored taint with effect NoSchedule then Kubernetes will not schedule the pod onto that node

• if there is no un-ignored taint with effect NoSchedule but there is at least one un-ignored taint with effect PreferNoSchedule then Kubernetes will try to not schedule the pod onto the node

• if there is at least one un-ignored taint with effect NoExecute then the pod will be evicted from the node (if it is already running on the node), and will not be scheduled onto the node (if it is not yet running on the node).

For example, imagine you taint a node like this

kubectl taint nodes node1 key1=value1:NoSchedule
kubectl taint nodes node1 key1=value1:NoExecute
kubectl taint nodes node1 key2=value2:NoSchedule

And a pod has two tolerations:

tolerations:
- key: "key1"
  operator: "Equal"
  value: "value1"
  effect: "NoSchedule"
- key: "key1"
  operator: "Equal"
  value: "value1"
  effect: "NoExecute"

In this case, the pod will not be able to schedule onto the node, because there is no toleration matching the third taint. But it will be able to continue running if it is already running on the node when the taint is added, because the third taint is the only one of the three that is not tolerated by the pod.

Normally, if a taint with effect NoExecute is added to a node, then any pods that do not tolerate the taint will be evicted immediately, and pods that do tolerate the taint will never be evicted. However, a toleration with NoExecute effect can specify an optional tolerationSeconds field that dictates how long the pod will stay bound to the node after the taint is added. For example,

tolerations:
- key: "key1"
  operator: "Equal"
  value: "value1"
  effect: "NoExecute"
  tolerationSeconds: 3600

means that if this pod is running and a matching taint is added to the node, then the pod will stay bound to the node for 3600 seconds, and then be evicted. If the taint is removed before that time, the pod will not be evicted.

Example Use Cases

Taints and tolerations are a flexible way to steer pods away from nodes or evict pods that shouldn't be running. A few of the use cases are

• Taint based Evictions: A per-pod-configurable eviction behavior when there are node problems, which is described in the next section.

Taint based Evictions

FEATURE STATE: Kubernetes v1.18 [stable]

The NoExecute taint effect, mentioned above, affects pods that are already running on the node as follows

• pods that do not tolerate the taint are evicted immediately

• pods that tolerate the taint without specifying tolerationSeconds in their toleration specification remain bound forever

• pods that tolerate the taint with a specified tolerationSeconds remain bound for the specified amount of time

The node controller automatically taints a Node when certain conditions are true. The following taints are built in:

• node.kubernetes.io/not-ready: Node is not ready. This corresponds to the NodeCondition Ready being "False".

• node.kubernetes.io/unreachable: Node is unreachable from the node controller. This corresponds to the NodeCondition Ready being "Unknown".

• node.kubernetes.io/memory-pressure: Node has memory pressure.

• node.kubernetes.io/disk-pressure: Node has disk pressure.

• node.kubernetes.io/pid-pressure: Node has PID pressure.

• node.kubernetes.io/network-unavailable: Node's network is unavailable.

• node.kubernetes.io/unschedulable: Node is unschedulable.

• node.cloudprovider.kubernetes.io/uninitialized: When the kubelet is started with "external" cloud provider, this taint is set on a node to mark it as unusable. After a controller from the cloud-controller-manager initializes this node, the kubelet removes this taint.

In case a node is to be evicted, the node controller or the kubelet adds relevant taints with NoExecute effect. If the fault condition returns to normal the kubelet or node controller can remove the relevant taint(s).

In some cases when the node is unreachable, the API server is unable to communicate with the kubelet on the node. The decision to delete the pods cannot be communicated to the kubelet until communication with the API server is re-established. In the meantime, the pods that are scheduled for deletion may continue to run on the partitioned node.

Note: The control plane limits the rate of adding node new taints to nodes. This rate limiting manages the number of evictions that are triggered when many nodes become unreachable at once (for example: if there is a network disruption).

You can specify tolerationSeconds for a Pod to define how long that Pod stays bound to a failing or unresponsive Node.

For example, you might want to keep an application with a lot of local state bound to node for a long time in the event of network partition, hoping that the partition will recover and thus the pod eviction can be avoided. The toleration you set for that Pod might look like:

tolerations:
- key: "node.kubernetes.io/unreachable"
  operator: "Exists"
  effect: "NoExecute"
  tolerationSeconds: 6000

Note:

Kubernetes automatically adds a toleration for node.kubernetes.io/not-ready and node.kubernetes.io/unreachable with tolerationSeconds=300, unless you, or a controller, set those tolerations explicitly.

These automatically-added tolerations mean that Pods remain bound to Nodes for 5 minutes after one of these problems is detected.

• node.kubernetes.io/unreachable

• node.kubernetes.io/not-ready

This ensures that DaemonSet pods are never evicted due to these problems.

Taint Nodes by Condition

The scheduler checks taints, not node conditions, when it makes scheduling decisions. This ensures that node conditions don't directly affect scheduling. For example, if the DiskPressure node condition is active, the control plane adds the node.kubernetes.io/disk-pressure taint and does not schedule new pods onto the affected node. If the MemoryPressure node condition is active, the control plane adds the node.kubernetes.io/memory-pressure taint.

The DaemonSet controller automatically adds the following NoSchedule tolerations to all daemons, to prevent DaemonSets from breaking.

• node.kubernetes.io/memory-pressure

• node.kubernetes.io/disk-pressure

• node.kubernetes.io/pid-pressure (1.14 or later)

• node.kubernetes.io/unschedulable (1.10 or later)

• node.kubernetes.io/network-unavailable (host network only)

Adding these tolerations ensures backward compatibility. You can also add arbitrary tolerations to DaemonSets.

Kubernetes Nodes need occasional maintenance. You could be updating the Node’s kernel, resizing its compute resource in your cloud account, or replacing physical hardware components in a self-hosted installation.

Kubernetes cordons and drains are two mechanisms you can use to safely prepare for Node downtime. They allow workloads running on a target Node to be rescheduled onto other ones. You can then shutdown the Node or remove it from your cluster without impacting service availability.

Applying a Node Cordon

Cordoning a Node marks it as unavailable to the Kubernetes scheduler. The Node will be ineligible to host any new Pods subsequently added to your cluster.

Use the kubectl cordon command to place a cordon around a named Node:

$ kubectl cordon node-1
node/node-1 cordoned

Existing Pods already running on the Node won’t be affected by the cordon. They’ll remain accessible and will still be hosted by the cordoned Node.

You can check which of your Nodes are currently cordoned with the get nodes command:

$ kubectl get nodes
NAME       STATUS                     ROLES                  AGE   VERSION
node-1     Ready,SchedulingDisabled   control-plane,master   26m   v1.23.3

Cordoned nodes appear with the SchedulingDisabled status.

Draining a Node

Run kubectl drain to initiate a drain procedure. Specify the name of the Node you’re taking out for maintenance:

$ kubectl drain node-1
node/node-1 already cordoned
evicting pod kube-system/storage-provisioner
evicting pod default/nginx-7c658794b9-zszdd
evicting pod kube-system/coredns-64897985d-dp6lx
pod/storage-provisioner evicted
pod/nginx-7c658794b9-zszdd evicted
pod/coredns-64897985d-dp6lx evicted
node/node-1 evicted

The drain procedure first cordons the Node if you’ve not already placed one manually. It will then evict running Kubernetes workloads by safely rescheduling them to other Nodes in your cluster.

You can shutdown or destroy the Node once the drain’s completed. You’ve freed the Node from its responsibilities to your cluster. The cordon provides an assurance that no new workloads have been scheduled since the drain completed.

Ignoring Pod Grace Periods

Drains can sometimes take a while to complete if your Pods have long grace periods. This might not be ideal when you need to urgently take a Node offline. Use the --grace-period flag to override Pod termination grace periods and force an immediate eviction:

$ kubectl drain node-1 --grace-period 0

This should be used with care – some workloads might not respond well if they’re stopped without being offered a chance to clean up.

Understanding the pod phase

pod’s conditions during its lifecycle

Understanding the container state

Configuring the pod’s restart policy

What are the Three Types of Kubernetes Probes?

Kubernetes provides the following types of probes. For all these types, if the container does not implement the probe handler, their result is always Success.

• Readiness Probe—indicates whether the application running in the container is ready to accept requests. If so, Services matching the pod are allowed to send traffic to it. If not, the endpoints controller removes the pod from all matching Kubernetes Services.

• Startup Probe—indicates whether the application running in the container has started. If so, other probes start functioning. If not, the kubelet kills and restarts the container.

When to Use Readiness Probes

Readiness probes are most useful when an application is temporarily malfunctioning and unable to serve traffic. If the application is running but not fully available, Kubernetes may not be able to scale it up and new deployments could fail. A readiness probe allows Kubernetes to wait until the service is active before sending it traffic.

When you use a readiness probe, keep in mind that Kubernetes will only send traffic to the pod if the probe succeeds.

There is no need to use a readiness probe on deletion of a pod. When a pod is deleted, it automatically puts itself into an unready state, regardless of whether readiness probes are used. It remains in this status until all containers in the pod have stopped.

How Readiness Probes Work in Kubernetes

A readiness probe can be deployed as part of several Kubernetes objects. For example, here is how to define a readiness probe in a Deployment:

Once the above Deployment object is applied to the cluster, the readiness probe runs continuously throughout the lifecycle of the application.

A readiness probe has the following configuration options:

Why Do Readiness Probes Fail? Common Error Scenarios

Readiness probes are used to verify tasks during a container lifecycle. This means that if the probe’s response is interrupted or delayed, service may be interrupted. Keep in mind that if a readiness probe returns Failure status, Kubernetes will remove the pod from all matching service endpoints. Here are two examples of conditions that can cause an application to incorrectly fail the readiness probe.

Delayed Response

In some circumstances, readiness probes may be late to respond—for example, if the application needs to read large amounts of data with low latency or perform heavy computations. Consider this behavior when configuring readiness probes, and always test your application thoroughly before running it in production with a readiness probe.

Cascading Failures

A readiness probe response can be conditional on components that are outside the direct control of the application. For example, you could configure a readiness probe using HTTPGet, in such a way that the application first checks the availability of a cache service or database before responding to the probe. This means that if the database is down or late to respond, the entire application will become unavailable.

This may or may not make sense, depending on your application setup. If the application cannot function at all without the third-party component, maybe this behavior is warranted. If it can continue functioning, for example, by falling back to a local cache, the database or external cache should not be connected to probe responses.

In general, if the pod is technically ready, even if it cannot function perfectly, it should not fail the readiness probe. A good compromise is to implement a “degraded mode,” for example, if there is no access to the database, answer read requests that can be addressed by local cache and return 503 (service unavailable) on write requests. Ensure that downstream services are resilient to a failure in the upstream service.

Methods of Checking Application Health

Startup, readiness, and liveness probes can check the health of applications in three ways: HTTP checks, container execution checks, and TCP socket checks.

HTTP Checks

An HTTP check is ideal for applications that return HTTP status codes, such as REST APIs.

HTTP probe uses GET requests to check the health of an application. The check is successful if the HTTP response code is in the range 200-399.

The following example demonstrates how to implement a readiness probe with the HTTP check method:

...contents omitted...
readinessProbe:
  httpGet:
    path: /health (1)
    port: 8080
  initialDelaySeconds: 15 (2)
  timeoutSeconds: 1 (3)
...contents omitted...

1. The readiness probe endpoint.

2. How long to wait after the container starts before checking its health.

3. How long to wait for the probe to finish.

Container Execution Checks

Container execution checks are ideal in scenarios where you must determine the status of the container based on the exit code of a process or shell script running in the container.

When using container execution checks Kubernetes executes a command inside the container. Exiting the check with a status of 0 is considered a success. All other status codes are considered a failure.

The following example demonstrates how to implement a container execution check:

...contents omitted...
livenessProbe:
  exec:
    command:(1)
    - cat
    - /tmp/health
  initialDelaySeconds: 15
  timeoutSeconds: 1
...contents omitted...

1. The command to run and its arguments, as a YAML array.

TCP Socket Checks

A TCP socket check is ideal for applications that run as daemons, and open TCP ports, such as database servers, file servers, web servers, and application servers.

When using TCP socket checks Kubernetes attempts to open a socket to the container. The container is considered healthy if the check can establish a successful connection.

The following example demonstrates how to implement a liveness probe by using the TCP socket check method:

...contents omitted...
livenessProbe:
  tcpSocket:
    port: 8080 (1)
  initialDelaySeconds: 15
  timeoutSeconds: 1
...contents omitted...

1. The TCP port to check.

Creating Probes

To configure probes on a deployment, edit the deployment’s resource definition. To do this, you can use the kubectl edit or kubectl patch commands. Alternatively, if you already have a deployment YAML definition, you can modify it to include the probes and then apply it with kubectl apply.

The following example demonstrates using the kubectl edit command to add a readiness probe to a deployment:

[user@host ~]$ kubectl set probe deployment myapp --readiness \
--get-url=http://:8080/healthz --period=20
[user@host ~]$ kubectl patch deployment myapp \
-p '{"spec":{"containers"[0]: {"readinessProbe": {}}}}'

The following examples demonstrate using the kubectl set probe command with a variety of options:

[user@host ~]$ kubectl set probe deployment myapp --readiness \
--get-url=http://:8080/healthz --period=20

[user@host ~]$ kubectl set probe deployment myapp --liveness \
--open-tcp=3306 --period=20 \
--timeout-seconds=1

[user@host ~]$ kubectl set probe deployment myapp --liveness \
--get-url=http://:8080/healthz --initial-delay-seconds=30 \
--success-threshold=1 --failure-threshold=3

Assign Pods to Nodes using Node Affinity

This page shows how to assign a Kubernetes Pod to a particular node using Node Affinity in a Kubernetes cluster.

Before you begin

Your Kubernetes server must be at or later than version v1.10. To check the version, enter kubectl version.

Add a label to a node

1. List the nodes in your cluster, along with their labels:

   Copy

   kubectl get nodes --show-labels

   The output is similar to this:

   Copy

   NAME      STATUS    ROLES    AGE     VERSION        LABELS
   worker0   Ready     <none>   1d      v1.13.0        ...,kubernetes.io/hostname=worker0
   worker1   Ready     <none>   1d      v1.13.0        ...,kubernetes.io/hostname=worker1
   worker2   Ready     <none>   1d      v1.13.0        ...,kubernetes.io/hostname=worker2

2. Choose one of your nodes, and add a label to it:

   Copy

   kubectl label nodes <your-node-name> disktype=ssd

   where <your-node-name> is the name of your chosen node.

3. Verify that your chosen node has a disktype=ssd label:

   Copy

   kubectl get nodes --show-labels

   The output is similar to this:

   Copy

   NAME      STATUS    ROLES    AGE     VERSION        LABELS
   worker0   Ready     <none>   1d      v1.13.0        ...,disktype=ssd,kubernetes.io/hostname=worker0
   worker1   Ready     <none>   1d      v1.13.0        ...,kubernetes.io/hostname=worker1
   worker2   Ready     <none>   1d      v1.13.0        ...,kubernetes.io/hostname=worker2

   In the preceding output, you can see that the worker0 node has a disktype=ssd label.

Schedule a Pod using required node affinity

This manifest describes a Pod that has a requiredDuringSchedulingIgnoredDuringExecution node affinity,disktype: ssd. This means that the pod will get scheduled only on a node that has a disktype=ssd label.

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: disktype
            operator: In
            values:
            - ssd            
  containers:
  - name: nginx
    image: nginx
    imagePullPolicy: IfNotPresent

1. Apply the manifest to create a Pod that is scheduled onto your chosen node:

   Copy

   kubectl apply -f https://k8s.io/examples/pods/pod-nginx-required-affinity.yaml

2. Verify that the pod is running on your chosen node:

   Copy

   kubectl get pods --output=wide

   The output is similar to this:

   Copy

   NAME     READY     STATUS    RESTARTS   AGE    IP           NODE
   nginx    1/1       Running   0          13s    10.200.0.4   worker0

Schedule a Pod using preferred node affinity

This manifest describes a Pod that has a preferredDuringSchedulingIgnoredDuringExecution node affinity,disktype: ssd. This means that the pod will prefer a node that has a disktype=ssd label.

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 1
        preference:
          matchExpressions:
          - key: disktype
            operator: In
            values:
            - ssd          
  containers:
  - name: nginx
    image: nginx
    imagePullPolicy: IfNotPresent

1. Apply the manifest to create a Pod that is scheduled onto your chosen node:

   Copy

   kubectl apply -f https://k8s.io/examples/pods/pod-nginx-preferred-affinity.yaml

2. Verify that the pod is running on your chosen node:

   Copy

   kubectl get pods --output=wide

   The output is similar to this:

   Copy

   NAME     READY     STATUS    RESTARTS   AGE    IP           NODE
   nginx    1/1       Running   0          13s    10.200.0.4   worker0

Lab 1 : Replicaset lab

Lab 2 : Create Statfulset for mySQl DB with PVC and Secret

Step 1: Create secret with password

• Step 1 : Encrypte password with base64

• Step 2: Create secret with password

• Step 3: Create PVC to store mysql db

• Step 4: Create Statefulset for MySql DB

• Step 5: Expose DB service port 3306

Amazon RDS for Oracle is a fully managed commercial database that makes it easy to set up, operate, and scale Oracle deployments in the cloud. Amazon RDS frees you up to focus on innovation and application development by managing time-consuming database administration tasks including provisioning, backups, software patching, monitoring, and hardware scaling. You can run Amazon RDS for Oracle under two different licensing models – “License Included” and “Bring-Your-Own-License (BYOL)”. In the "License Included" service model, you do not need separately purchased Oracle licenses; the Oracle Database software has been licensed by Amazon Web Services.

You can take advantage of hourly pricing with no upfront fees or long-term commitments. In addition, you also have the option to purchase Reserved DB Instances under one or three year reservation terms. With Reserved DB Instances, you can make low, one-time, upfront payment for each DB Instance and then pay a significantly discounted hourly usage rate, achieving up to 48% net cost savings.

Benefits

Fully managed

Amazon RDS for Oracle is fully managed by Amazon Relational Database Service (RDS). You no longer need to worry about database management tasks such as hardware provisioning, software patching, setup, configuration, or backups.

Availability and reliability

Amazon RDS for Oracle makes it easy to use replication to enhance availability and reliability for production workloads. Using the Multi-AZ deployment option you can run mission critical workloads with high availability and built-in automated fail-over from your primary database to a synchronously replicated secondary database in case of a failure.

Ease of migration

Amazon RDS for Oracle gives you the full benefits of a managed service solution. You can use the lift-and-shift approach to migrate your legacy Oracle database to Amazon RDS for Oracle and, as a result, reduce the need to refactor and change existing application components.

Fast, predictable performance

Amazon RDS instances supports up to 64 TiB of storage built on high-performance Amazon Elastic Block Store (EBS) SSD volumes. Amazon RDS General Purpose (SSD) storage delivers a consistent baseline of 3 IOPS per provisioned GB and provides the ability to burst up to 12,000 IOPS. With provisioned IOPS, you can provision up to 256,000 IOPS per database instance.

License flexibility

You can run Amazon RDS for Oracle under two different licensing models - "License Included" and "Bring-Your-Own-License (BYOL)". In the "License Included" service model, you do not need separately purchased Oracle licenses.

Multiple availability zones

Using the Multi-AZ deployment option you can run mission critical workloads with high availability and built-in automated fail-over from your primary database to a synchronously replicated secondary database in case of a failure. As with all Amazon Web Services, there are no up-front investments required, and you pay only for the resources you use.

SQL Server is a relational database management system developed by Microsoft. Amazon RDS for SQL Server makes it easy to set up, operate, and scale SQL Server deployments in the cloud. With Amazon RDS, you can deploy multiple editions of SQL Server (2012, 2014, 2016, 2017 and 2019) including Express, Web, Standard and Enterprise, in minutes with cost-efficient and re-sizable compute capacity. Amazon RDS frees you up to focus on application development by managing time-consuming database administration tasks including provisioning, backups, software patching, monitoring, and hardware scaling.

Amazon RDS for SQL Server supports the “License Included” licensing model. You do not need separately purchased Microsoft SQL Server licenses. "License Included" pricing is inclusive of software, underlying hardware resources, and Amazon RDS management capabilities.

You can take advantage of hourly pricing with no upfront fees or long-term commitments. In addition, you also have the option to purchase Reserved DB Instances under one or three year reservation terms. With Reserved DB Instances, you can make low, one-time, upfront payment for each DB Instance and then pay a significantly discounted hourly usage rate, achieving up to 65% net cost savings.

Amazon RDS for SQL Server DB Instances can be provisioned with either standard storage or Provisioned IOPS storage. Amazon RDS Provisioned IOPS is a storage option designed to deliver fast, predictable, and consistent I/O performance, and is optimized for I/O-intensive, transactional (OLTP) database workloads.

Benefits

Fully managed

Amazon RDS for SQL Server is fully managed by Amazon Relational Database Service (RDS). You no longer need to worry about database management tasks such as hardware provisioning, software patching, setup, configuration, or backups.

Single click high availability

With a single click, you can enable the Multi-AZ option, replicating data synchronously across different availability zones. In case the primary node crashes, your database will automatically fail over to the secondary and we will automatically re-build the secondary.

Auto-scaled storage

By opting into Auto-Scale storage, instances will automatically increase the storage size with zero downtime. With RDS Storage Auto Scaling, you simply set your desired maximum storage limit, and Auto Scaling takes care of the rest.

Automated backups

Amazon RDS creates and saves automated backups of your SQL Server instance. Amazon RDS creates a storage volume snapshot of your instance, backing up the entire instance and not just individual databases. Amazon RDS for SQL Server creates automated backups of your DB instance during the backup window of your DB instance.

Upgrade and modernize

Amazon RDS SQL Server offers Enterprise, Standard Edition, Web, and Express Editions for SQL Server Versions 2012, 2014, 2016, 2017, and 2019. For SQL Server 2008 R2 (now deprecated) customers can still keep their database compatibility in 2008 and migrate to a new versions with minimal changes.

Ease of migration

We support a number of ways to migrate to Amazon RDS SQL Server, including Single and Multi-file native restores, Microsoft SQL Server Database Publishing Wizard, Import/Export, Amazon Database Migration Service, and SQL Server Replication.

Amazon RDS on VMware provides Amazon's relational database service for on-premises VMware infrastructure and points to the potential future of hybrid cloud services.

Many people used to see VMware and AWS as bitter enemies, but that changed when they announced VMware Cloud on AWS in 2017. The announcement of Amazon RDS on VMware at VMworld 2018 indicated an evolving relationship between the two companies that could point to potential. For greater collaboration between cloud vendors and on-premises infrastructure.

The investment required to move everything to the cloud has proved too expensive for many organizations, so as realistic expectations are set, more and more cloud vendors will try to integrate the features and benefits of the cloud with their on-premises infrastructure.

Amazon RDS aims to make it easy to set up, run and develop relational databases. RDS provides a flexible and easy way to do previously tedious tasks like patching, capacity management and database tuning. Because this is an Amazon service, it only offers on-demand pricing for what you use or pay for reserved, dedicated capacity. Until recently, Amazon limited this service to Amazon Cloud.

Amazon RDS on VMware is in Technical Preview, so all the details about how the platform works are currently unavailable. If it's anything like the native Amazon RDS, you'll be able to create and manage databases from half a dozen popular database types, including Oracle and Microsoft SQL Server.

Amazon RDS for VMware will enable the affordable, high-availability hybrid deployment, simple database disaster recovery for AWS, and read-only clones of on-premises data in AWS. This partnership could help Amazon customers easily migrate traditional database deployment from their sites and AWS, even sites with difficult licensing requirements. It can also help VMware customers see the benefits of the AWS management stack for databases in traditional infrastructure.

Amazon RDS Vs. VMware

Amazon has focused on moving workloads to the public cloud; Even VMware Cloud on AWS has focused on moving older workloads off-premises. Amazon RDS on VMware is different. With this release, Amazon is following Microsoft to provide public cloud services within the confines of the data centre.

Let's say Amazon continues to add its services to the data centre and continues to do the same with Microsoft Azure Stack. In that case, customers can see several major cloud benefits without the enormous effort of moving workloads to the public cloud. As much as the industry has talked about hybrid cloud, most customers haven't implemented it.

It takes a major investment for customers to achieve the flexibility, ease of consumption, agility and scale of the public cloud. Instead of making that investment, organizations often just put the time and effort into moving particular workloads to the cloud. As cloud providers look to expand the public cloud into a private data centre, customers can get the best of both worlds and have a true hybrid cloud model.

The ability to run Amazon services inside your data centre is all but Amazon admitting that the public cloud isn't the only way forward. Combine that with the fact that you can now run a traditional VMware on-premises infrastructure inside Amazon's public cloud, and you can see that the two companies have decided to have public, private, and hybrid clouds that all coexist.

A time-to-live mechanism to clean up old Jobs that have finished execution.

ReplicaSet

A ReplicaSet's purpose is to maintain a stable set of replica Pods running at any given time. As such, it is often used to guarantee the availability of a specified number of identical Pods.

How a ReplicaSet works

A ReplicaSet is defined with fields, including a selector that specifies how to identify Pods it can acquire, a number of replicas indicating how many Pods it should be maintaining, and a pod template specifying the data of new Pods it should create to meet the number of replicas criteria. A ReplicaSet then fulfills its purpose by creating and deleting Pods as needed to reach the desired number. When a ReplicaSet needs to create new Pods, it uses its Pod template.

When to use a ReplicaSet

A ReplicaSet ensures that a specified number of pod replicas are running at any given time. However, a Deployment is a higher-level concept that manages ReplicaSets and provides declarative updates to Pods along with a lot of other useful features. Therefore, we recommend using Deployments instead of directly using ReplicaSets, unless you require custom update orchestration or don't require updates at all.

This actually means that you may never need to manipulate ReplicaSet objects: use a Deployment instead, and define your application in the spec section.

Example

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: frontend
  labels:
    app: guestbook
    tier: frontend
spec:
  # modify replicas according to your case
  replicas: 3
  selector:
    matchLabels:
      tier: frontend
  template:
    metadata:
      labels:
        tier: frontend
    spec:
      containers:
      - name: php-redis
        image: gcr.io/google_samples/gb-frontend:v3

Saving this manifest into frontend.yaml and submitting it to a Kubernetes cluster will create the defined ReplicaSet and the Pods that it manages.

kubectl apply -f https://kubernetes.io/examples/controllers/frontend.yaml

You can then get the current ReplicaSets deployed:

kubectl get rs

And see the frontend one you created:

NAME       DESIRED   CURRENT   READY   AGE
frontend   3         3         3       6s

You can also check on the state of the ReplicaSet:

kubectl describe rs/frontend

And you will see output similar to:

Name:         frontend
Namespace:    default
Selector:     tier=frontend
Labels:       app=guestbook
              tier=frontend
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"apps/v1","kind":"ReplicaSet","metadata":{"annotations":{},"labels":{"app":"guestbook","tier":"frontend"},"name":"frontend",...
Replicas:     3 current / 3 desired
Pods Status:  3 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:  tier=frontend
  Containers:
   php-redis:
    Image:        gcr.io/google_samples/gb-frontend:v3
    Port:         <none>
    Host Port:    <none>
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Events:
  Type    Reason            Age   From                   Message
  ----    ------            ----  ----                   -------
  Normal  SuccessfulCreate  117s  replicaset-controller  Created pod: frontend-wtsmm
  Normal  SuccessfulCreate  116s  replicaset-controller  Created pod: frontend-b2zdv
  Normal  SuccessfulCreate  116s  replicaset-controller  Created pod: frontend-vcmts

And lastly you can check for the Pods brought up:

kubectl get pods

You should see Pod information similar to:

NAME             READY   STATUS    RESTARTS   AGE
frontend-b2zdv   1/1     Running   0          6m36s
frontend-vcmts   1/1     Running   0          6m36s
frontend-wtsmm   1/1     Running   0          6m36s

You can also verify that the owner reference of these pods is set to the frontend ReplicaSet. To do this, get the yaml of one of the Pods running:

kubectl get pods frontend-b2zdv -o yaml

The output will look similar to this, with the frontend ReplicaSet's info set in the metadata's ownerReferences field:

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2020-02-12T07:06:16Z"
  generateName: frontend-
  labels:
    tier: frontend
  name: frontend-b2zdv
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: frontend
    uid: f391f6db-bb9b-4c09-ae74-6a1f77f3d5cf
...

Non-Template Pod acquisitions

While you can create bare Pods with no problems, it is strongly recommended to make sure that the bare Pods do not have labels which match the selector of one of your ReplicaSets. The reason for this is because a ReplicaSet is not limited to owning Pods specified by its template-- it can acquire other Pods in the manner specified in the previous sections.

Take the previous frontend ReplicaSet example, and the Pods specified in the following manifest:

apiVersion: v1
kind: Pod
metadata:
  name: pod1
  labels:
    tier: frontend
spec:
  containers:
  - name: hello1
    image: gcr.io/google-samples/hello-app:2.0

---

apiVersion: v1
kind: Pod
metadata:
  name: pod2
  labels:
    tier: frontend
spec:
  containers:
  - name: hello2
    image: gcr.io/google-samples/hello-app:1.0

As those Pods do not have a Controller (or any object) as their owner reference and match the selector of the frontend ReplicaSet, they will immediately be acquired by it.

Suppose you create the Pods after the frontend ReplicaSet has been deployed and has set up its initial Pod replicas to fulfill its replica count requirement:

kubectl apply -f https://kubernetes.io/examples/pods/pod-rs.yaml

The new Pods will be acquired by the ReplicaSet, and then immediately terminated as the ReplicaSet would be over its desired count.

Fetching the Pods:

kubectl get pods

The output shows that the new Pods are either already terminated, or in the process of being terminated:

NAME             READY   STATUS        RESTARTS   AGE
frontend-b2zdv   1/1     Running       0          10m
frontend-vcmts   1/1     Running       0          10m
frontend-wtsmm   1/1     Running       0          10m
pod1             0/1     Terminating   0          1s
pod2             0/1     Terminating   0          1s

If you create the Pods first:

kubectl apply -f https://kubernetes.io/examples/pods/pod-rs.yaml

And then create the ReplicaSet however:

kubectl apply -f https://kubernetes.io/examples/controllers/frontend.yaml

You shall see that the ReplicaSet has acquired the Pods and has only created new ones according to its spec until the number of its new Pods and the original matches its desired count. As fetching the Pods:

kubectl get pods

Will reveal in its output:

NAME             READY   STATUS    RESTARTS   AGE
frontend-hmmj2   1/1     Running   0          9s
pod1             1/1     Running   0          36s
pod2             1/1     Running   0          36s

In this manner, a ReplicaSet can own a non-homogenous set of Pods

Writing a ReplicaSet manifest

As with all other Kubernetes API objects, a ReplicaSet needs the apiVersion, kind, and metadata fields. For ReplicaSets, the kind is always a ReplicaSet.

Pod Template

Pod Selector

matchLabels:
  tier: frontend

In the ReplicaSet, .spec.template.metadata.labels must match spec.selector, or it will be rejected by the API.

Note: For 2 ReplicaSets specifying the same .spec.selector but different .spec.template.metadata.labels and .spec.template.spec fields, each ReplicaSet ignores the Pods created by the other ReplicaSet.

Replicas

You can specify how many Pods should run concurrently by setting .spec.replicas. The ReplicaSet will create/delete its Pods to match this number.

If you do not specify .spec.replicas, then it defaults to 1.

Working with ReplicaSets

Deleting a ReplicaSet and its Pods

When using the REST API or the client-go library, you must set propagationPolicy to Background or Foreground in the -d option. For example:

kubectl proxy --port=8080
curl -X DELETE  'localhost:8080/apis/apps/v1/namespaces/default/replicasets/frontend' \
  -d '{"kind":"DeleteOptions","apiVersion":"v1","propagationPolicy":"Foreground"}' \
  -H "Content-Type: application/json"

Deleting just a ReplicaSet

kubectl proxy --port=8080
curl -X DELETE  'localhost:8080/apis/apps/v1/namespaces/default/replicasets/frontend' \
  -d '{"kind":"DeleteOptions","apiVersion":"v1","propagationPolicy":"Orphan"}' \
  -H "Content-Type: application/json"

Isolating Pods from a ReplicaSet

You can remove Pods from a ReplicaSet by changing their labels. This technique may be used to remove Pods from service for debugging, data recovery, etc. Pods that are removed in this way will be replaced automatically ( assuming that the number of replicas is not also changed).

Scaling a ReplicaSet

A ReplicaSet can be easily scaled up or down by simply updating the .spec.replicas field. The ReplicaSet controller ensures that a desired number of Pods with a matching label selector are available and operational.

When scaling down, the ReplicaSet controller chooses which pods to delete by sorting the available pods to prioritize scaling down pods based on the following general algorithm:

1. Pending (and unschedulable) pods are scaled down first

2. If controller.kubernetes.io/pod-deletion-cost annotation is set, then the pod with the lower value will come first.

3. Pods on nodes with more replicas come before pods on nodes with fewer replicas.

If all of the above match, then selection is random.

Pod deletion cost

FEATURE STATE: Kubernetes v1.22 [beta]

The annotation should be set on the pod, the range is [-2147483647, 2147483647]. It represents the cost of deleting a pod compared to other pods belonging to the same ReplicaSet. Pods with lower deletion cost are preferred to be deleted before pods with higher deletion cost.

The implicit value for this annotation for pods that don't set it is 0; negative values are permitted. Invalid values will be rejected by the API server.

Note:

• This is honored on a best-effort basis, so it does not offer any guarantees on pod deletion order.

• Users should avoid updating the annotation frequently, such as updating it based on a metric value, because doing so will generate a significant number of pod updates on the apiserver.

Example Use Case

The different pods of an application could have different utilization levels. On scale down, the application may prefer to remove the pods with lower utilization. To avoid frequently updating the pods, the application should update controller.kubernetes.io/pod-deletion-cost once before issuing a scale down (setting the annotation to a value proportional to pod utilization level). This works if the application itself controls the down scaling; for example, the driver pod of a Spark deployment.

ReplicaSet as a Horizontal Pod Autoscaler Target

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: frontend-scaler
spec:
  scaleTargetRef:
    kind: ReplicaSet
    name: frontend
  minReplicas: 3
  maxReplicas: 10
  targetCPUUtilizationPercentage: 50

Saving this manifest into hpa-rs.yaml and submitting it to a Kubernetes cluster should create the defined HPA that autoscales the target ReplicaSet depending on the CPU usage of the replicated Pods.

kubectl apply -f https://k8s.io/examples/controllers/hpa-rs.yaml

Alternatively, you can use the kubectl autoscale command to accomplish the same (and it's easier!)

kubectl autoscale rs frontend --max=10 --min=3 --cpu-percent=50

ReplicationController

A ReplicationController ensures that a specified number of pod replicas are running at any one time. In other words, a ReplicationController makes sure that a pod or a homogeneous set of pods is always up and available.

How a ReplicationController Works

If there are too many pods, the ReplicationController terminates the extra pods. If there are too few, the ReplicationController starts more pods. Unlike manually created pods, the pods maintained by a ReplicationController are automatically replaced if they fail, are deleted, or are terminated. For example, your pods are re-created on a node after disruptive maintenance such as a kernel upgrade. For this reason, you should use a ReplicationController even if your application requires only a single pod. A ReplicationController is similar to a process supervisor, but instead of supervising individual processes on a single node, the ReplicationController supervises multiple pods across multiple nodes.

ReplicationController is often abbreviated to "rc" in discussion, and as a shortcut in kubectl commands.

A simple case is to create one ReplicationController object to reliably run one instance of a Pod indefinitely. A more complex use case is to run several identical replicas of a replicated service, such as web servers.

Running an example ReplicationController

This example ReplicationController config runs three copies of the nginx web server.

apiVersion: v1
kind: ReplicationController
metadata:
  name: nginx
spec:
  replicas: 3
  selector:
    app: nginx
  template:
    metadata:
      name: nginx
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80

Run the example job by downloading the example file and then running this command:

kubectl apply -f https://k8s.io/examples/controllers/replication.yaml

The output is similar to this:

replicationcontroller/nginx created

Check on the status of the ReplicationController using this command:

kubectl describe replicationcontrollers/nginx

The output is similar to this:

Name:        nginx
Namespace:   default
Selector:    app=nginx
Labels:      app=nginx
Annotations:    <none>
Replicas:    3 current / 3 desired
Pods Status: 0 Running / 3 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:       app=nginx
  Containers:
   nginx:
    Image:              nginx
    Port:               80/TCP
    Environment:        <none>
    Mounts:             <none>
  Volumes:              <none>
Events:
  FirstSeen       LastSeen     Count    From                        SubobjectPath    Type      Reason              Message
  ---------       --------     -----    ----                        -------------    ----      ------              -------
  20s             20s          1        {replication-controller }                    Normal    SuccessfulCreate    Created pod: nginx-qrm3m
  20s             20s          1        {replication-controller }                    Normal    SuccessfulCreate    Created pod: nginx-3ntk0
  20s             20s          1        {replication-controller }                    Normal    SuccessfulCreate    Created pod: nginx-4ok8v

Here, three pods are created, but none is running yet, perhaps because the image is being pulled. A little later, the same command may show:

Pods Status:    3 Running / 0 Waiting / 0 Succeeded / 0 Failed

To list all the pods that belong to the ReplicationController in a machine readable form, you can use a command like this:

pods=$(kubectl get pods --selector=app=nginx --output=jsonpath={.items..metadata.name})
echo $pods

The output is similar to this:

nginx-3ntk0 nginx-4ok8v nginx-qrm3m

Here, the selector is the same as the selector for the ReplicationController (seen in the kubectl describe output), and in a different form in replication.yaml. The --output=jsonpath option specifies an expression with the name from each pod in the returned list.

Writing a ReplicationController Manifest

As with all other Kubernetes config, a ReplicationController needs apiVersion, kind, and metadata fields.

Pod Template

The .spec.template is the only required field of the .spec.

Labels on the ReplicationController

The ReplicationController can itself have labels (.metadata.labels). Typically, you would set these the same as the .spec.template.metadata.labels; if .metadata.labels is not specified then it defaults to .spec.template.metadata.labels. However, they are allowed to be different, and the .metadata.labels do not affect the behavior of the ReplicationController.

Pod Selector

If specified, the .spec.template.metadata.labels must be equal to the .spec.selector, or it will be rejected by the API. If .spec.selector is unspecified, it will be defaulted to .spec.template.metadata.labels.

Also you should not normally create any pods whose labels match this selector, either directly, with another ReplicationController, or with another controller such as Job. If you do so, the ReplicationController thinks that it created the other pods. Kubernetes does not stop you from doing this.

Multiple Replicas

You can specify how many pods should run concurrently by setting .spec.replicas to the number of pods you would like to have running concurrently. The number running at any time may be higher or lower, such as if the replicas were just increased or decreased, or if a pod is gracefully shutdown, and a replacement starts early.

If you do not specify .spec.replicas, then it defaults to 1.

Working with ReplicationControllers

Deleting a ReplicationController and its Pods

Deleting only a ReplicationController

You can delete a ReplicationController without affecting any of its pods.

Isolating pods from a ReplicationController

Pods may be removed from a ReplicationController's target set by changing their labels. This technique may be used to remove pods from service for debugging and data recovery. Pods that are removed in this way will be replaced automatically (assuming that the number of replicas is not also changed).

Common usage patterns

Rescheduling

As mentioned above, whether you have 1 pod you want to keep running, or 1000, a ReplicationController will ensure that the specified number of pods exists, even in the event of node failure or pod termination (for example, due to an action by another control agent).

Scaling

The ReplicationController enables scaling the number of replicas up or down, either manually or by an auto-scaling control agent, by updating the replicas field.

Rolling updates

The ReplicationController is designed to facilitate rolling updates to a service by replacing pods one-by-one.

Ideally, the rolling update controller would take application readiness into account, and would ensure that a sufficient number of pods were productively serving at any given time.

The two ReplicationControllers would need to create pods with at least one differentiating label, such as the image tag of the primary container of the pod, since it is typically image updates that motivate rolling updates.

Multiple release tracks

In addition to running multiple releases of an application while a rolling update is in progress, it's common to run multiple releases for an extended period of time, or even continuously, using multiple release tracks. The tracks would be differentiated by labels.

For instance, a service might target all pods with tier in (frontend), environment in (prod). Now say you have 10 replicated pods that make up this tier. But you want to be able to 'canary' a new version of this component. You could set up a ReplicationController with replicas set to 9 for the bulk of the replicas, with labels tier=frontend, environment=prod, track=stable, and another ReplicationController with replicas set to 1 for the canary, with labels tier=frontend, environment=prod, track=canary. Now the service is covering both the canary and non-canary pods. But you can mess with the ReplicationControllers separately to test things out, monitor the results, etc.

Using ReplicationControllers with Services

Multiple ReplicationControllers can sit behind a single service, so that, for example, some traffic goes to the old version, and some goes to the new version.

A ReplicationController will never terminate on its own, but it isn't expected to be as long-lived as services. Services may be composed of pods controlled by multiple ReplicationControllers, and it is expected that many ReplicationControllers may be created and destroyed over the lifetime of a service (for instance, to perform an update of pods that run the service). Both services themselves and their clients should remain oblivious to the ReplicationControllers that maintain the pods of the services.

Writing programs for Replication

Responsibilities of the ReplicationController

Deployments

Note: Do not manage ReplicaSets owned by a Deployment. Consider opening an issue in the main Kubernetes repository if your use case is not covered below.

Use Case

The following are typical use cases for Deployments:

Creating a Deployment

The following is an example of a Deployment. It creates a ReplicaSet to bring up three nginx Pods:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

In this example:

• The Deployment creates a ReplicaSet that creates three replicated Pods, indicated by the .spec.replicas field.

• The .spec.selector field defines how the created ReplicaSet finds which Pods to manage. In this case, you select a label that is defined in the Pod template (app: nginx). However, more sophisticated selection rules are possible, as long as the Pod template itself satisfies the rule.

  Note: The .spec.selector.matchLabels field is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". All of the requirements, from both matchLabels and matchExpressions, must be satisfied in order to match.

• The template field contains the following sub-fields:

  • The Pods are labeled app: nginxusing the .metadata.labels field.

  • Create one container and name it nginx using the .spec.template.spec.containers[0].name field.

Before you begin, make sure your Kubernetes cluster is up and running. Follow the steps given below to create the above Deployment:

1. Create the Deployment by running the following command:

   Copy

   kubectl apply -f https://k8s.io/examples/controllers/nginx-deployment.yaml

2. Run kubectl get deployments to check if the Deployment was created.

   If the Deployment is still being created, the output is similar to the following:

   Copy

   NAME               READY   UP-TO-DATE   AVAILABLE   AGE
   nginx-deployment   0/3     0            0           1s

   When you inspect the Deployments in your cluster, the following fields are displayed:

   • NAME lists the names of the Deployments in the namespace.

   • READY displays how many replicas of the application are available to your users. It follows the pattern ready/desired.

   • UP-TO-DATE displays the number of replicas that have been updated to achieve the desired state.

   • AVAILABLE displays how many replicas of the application are available to your users.

   • AGE displays the amount of time that the application has been running.

   Notice how the number of desired replicas is 3 according to .spec.replicas field.

3. To see the Deployment rollout status, run kubectl rollout status deployment/nginx-deployment.

   The output is similar to:

   Copy

   Waiting for rollout to finish: 2 out of 3 new replicas have been updated...
   deployment "nginx-deployment" successfully rolled out

4. Run the kubectl get deployments again a few seconds later. The output is similar to this:

   Copy

   NAME               READY   UP-TO-DATE   AVAILABLE   AGE
   nginx-deployment   3/3     3            3           18s

   Notice that the Deployment has created all three replicas, and all replicas are up-to-date (they contain the latest Pod template) and available.

5. To see the ReplicaSet (rs) created by the Deployment, run kubectl get rs. The output is similar to this:

   Copy

   NAME                          DESIRED   CURRENT   READY   AGE
   nginx-deployment-75675f5897   3         3         3       18s

   ReplicaSet output shows the following fields:

   • NAME lists the names of the ReplicaSets in the namespace.

   • DESIRED displays the desired number of replicas of the application, which you define when you create the Deployment. This is the desired state.

   • CURRENT displays how many replicas are currently running.

   • READY displays how many replicas of the application are available to your users.

   • AGE displays the amount of time that the application has been running.

   Notice that the name of the ReplicaSet is always formatted as [DEPLOYMENT-NAME]-[HASH]. This name will become the basis for the Pods which are created.

   The HASH string is the same as the pod-template-hash label on the ReplicaSet.

6. To see the labels automatically generated for each Pod, run kubectl get pods --show-labels. The output is similar to:

   Copy

   NAME                                READY     STATUS    RESTARTS   AGE       LABELS
   nginx-deployment-75675f5897-7ci7o   1/1       Running   0          18s       app=nginx,pod-template-hash=75675f5897
   nginx-deployment-75675f5897-kzszj   1/1       Running   0          18s       app=nginx,pod-template-hash=75675f5897
   nginx-deployment-75675f5897-qqcnn   1/1       Running   0          18s       app=nginx,pod-template-hash=75675f5897

   The created ReplicaSet ensures that there are three nginx Pods.

Note:

You must specify an appropriate selector and Pod template labels in a Deployment (in this case, app: nginx).

Do not overlap labels or selectors with other controllers (including other Deployments and StatefulSets). Kubernetes doesn't stop you from overlapping, and if multiple controllers have overlapping selectors those controllers might conflict and behave unexpectedly.

Pod-template-hash label

Caution: Do not change this label.

The pod-template-hash label is added by the Deployment controller to every ReplicaSet that a Deployment creates or adopts.

This label ensures that child ReplicaSets of a Deployment do not overlap. It is generated by hashing the PodTemplate of the ReplicaSet and using the resulting hash as the label value that is added to the ReplicaSet selector, Pod template labels, and in any existing Pods that the ReplicaSet might have.

Updating a Deployment

Note: A Deployment's rollout is triggered if and only if the Deployment's Pod template (that is, .spec.template) is changed, for example if the labels or container images of the template are updated. Other updates, such as scaling the Deployment, do not trigger a rollout.

Follow the steps given below to update your Deployment:

1. Let's update the nginx Pods to use the nginx:1.16.1 image instead of the nginx:1.14.2 image.

   Copy

   kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.16.1

   or use the following command:

   Copy

   kubectl set image deployment/nginx-deployment nginx=nginx:1.16.1

   where deployment/nginx-deployment indicates the Deployment, nginx indicates the Container the update will take place and nginx:1.16.1 indicates the new image and its tag.

   The output is similar to:

   Copy

   deployment.apps/nginx-deployment image updated

   Alternatively, you can edit the Deployment and change .spec.template.spec.containers[0].image from nginx:1.14.2 to nginx:1.16.1:

   Copy

   kubectl edit deployment/nginx-deployment

   The output is similar to:

   Copy

   deployment.apps/nginx-deployment edited

2. To see the rollout status, run:

   Copy

   kubectl rollout status deployment/nginx-deployment

   The output is similar to this:

   Copy

   Waiting for rollout to finish: 2 out of 3 new replicas have been updated...

   or

   Copy

   deployment "nginx-deployment" successfully rolled out