K8S on CentOS7 Part I

System Info

Name	OS Version	Spec
master-node	CentOS Linux release 7.8.2003 (Core)	4 vCPU 16 GiB
worker-node1	CentOS Linux release 7.8.2003 (Core)	4 vCPU 16 GiB
worker-node2	CentOS Linux release 7.8.2003 (Core)	4 vCPU 16 GiB

Verify the MAC address and product_uuid are unique for every node

1. use ip link or ifconfig -a

ip link | awk '/link\/ether/ {print $2}'

or

ifconfig -a | awk '/ether/ {print $2}'

2. use sudo cat /sys/class/dmi/id/product_uuid

Letting iptables see bridged traffic

1. Check br_netfilter module is loaded.

lsmod | grep br_netfilter

# if not loaded, please run this command to load br_netfilter module
sudo modprobe br_netfilter

2. As requirement for your Linux Node's iptables to correctly see bridged traffic, you should ensure net.bridge.bridge-nf-call-iptables is set to 1 in your sysctl config

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sudo sysctl --system

Check required ports

• Control-plane Node(s)

  Protocol	Directon	Port Range	Purpose	Used By
  TCP	Inbound	6443	Kubernetes API server	All
  TCP	Inbound	2379-2380	etcd server client API	kube-apiserver, etcd
  TCP	Inbound	10250	Kubelet API	Self, Control plane
  TCP	Inbound	10251	kube-scheduler	Self
  TCP	Inbound	10252	kube-controller-mananger	Self

• Worker Node(s)

  Protocol	Directon	Port Range	Purpose	Used By
  TCP	Inbound	10250	Kubelet API	Self, Control plane
  TCP	Inbound	30000-32767	NodePort Services†	All

So let's setup firewalld to allow requriement ports

• On Master Node(s)

firewall-cmd --permanent --add-port=6443/tcp
firewall-cmd --permanent --add-port=2379-2380/tcp
firewall-cmd --permanent --add-port=10250/tcp
firewall-cmd --permanent --add-port=10251/tcp
firewall-cmd --permanent --add-port=10252/tcp

• On Worker Node(s)

firewall-cmd --permanent --add-port=10250/tcp
firewall-cmd --permanent --add-port=30000-32767/tcp

• Make the new settings persistent

firewall-cmd --runtime-to-permanent

• Restart firewalld.service

systemctl restart firewalld.service

• Check iptables rules

firewall-cmd --list-all

or

iptables -xvn -L

Disable swap

swapoff -a 
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

Install runtime

Kubernets uses a container runtime

On Linux Nodes, Kubernetes supports several container runtimes: Docker, containerd, CRI-O

If you don't specify a runtime, kubeadm automatically tries to detect an installed container runtime by scanning through a list of well known Unix domain sockets. The following table lists container runtimes and their associated socket paths:

Runtime	Path to Unix domian socket
Docker	/var/run/docker.sock
containerd	/run/containerd/containerd.sock
CRI-O	/var/run/crio/crio.sock

so let's install container runtime, in this case I'll use Docker

Install Docker CE

1. Install required packeages

   yum install -y yum-utils device-mapper-persistent-data lvm2
   

2. Add the Docker repository

   yum-config-manager --add-repo \
   https://download.docker.com/linux/centos/docker-ce.repo
   

3. Install Docker CE

   yum update -y && yum install -y \
   containerd.io-1.2.13 \
   docker-ce-19.03.11 \
   docker-ce-cli-19.03.11
   

4. Create /etc/docker

   mkdir /etc/docker
   

5. Set up the Docker daemon

   cat > /etc/docker/daemon.json <<EOF
   {
     "exec-opts": ["native.cgroupdriver=systemd"],
     "log-driver": "json-file",
     "log-opts": {
       "max-size": "100m"
     },
     "storage-driver": "overlay2",
     "storage-opts": [
       "overlay2.override_kernel_check=true"
     ]
   }
   EOF
   

   mkdir -p /etc/systemd/system/docker.service.d
   

6. Restart Docker

   systemctl daemon-reload
   systemctl restart docker
   

• Installing kubeadm, kubelet and kubectl

  We have to install below required pacakges.

  1. kubeadm：the command to bootstrap the cluster.
  2. kubelet：the component that runs on all of the machines in your cluster and does things like starting pods and containers.
  3. kubectl：the command line util to talk to your cluster.

kubeadm will not install or manage kubelet or kubectl for you, so you will need to ensure they match the version of the Kubernetes control plane you want kubeadm to install for you.

Intstall Kubernetes repository

cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF

Set SELinux in disabled mode

setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

Install kubelet kubeadm kubectl

yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
systemctl enable --now kubelet

• Configure cgroup driver used by kubelet on control-plane node

When using Docker, kubeadm will automatically detect the cgroup driver for the kubelet and set it in the /var/lib/kubelet/config.yaml file during runtime.