Bootstrapping Kubernetes cluster with kubeadm

Specification

• 3 nodes – ubuntu 22.04 installed
  • node 0: 192.168.1.150
  • node 1: 192.168.1.151
  • node 2: 192.168.1.152

Update system on all nodes

$ apt-get update && apt-get upgrade
$ apt install --yes \
		net-tools \
		socat \
		conntrack \
                ipvsadm \
		apt-transport-https \
		ca-certificates \
		curl

$ cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

$ modprobe overlay
$ modprobe br_netfilter

$ cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

$ sysctl --system

Install Containerd on all nodes

$ wget https://github.com/containerd/containerd/releases/download/v1.6.19/containerd-1.6.19-linux-amd64.tar.gz
$ tar Cxzvf /usr/local containerd-1.6.19-linux-amd64.tar.gz

$ wget https://raw.githubusercontent.com/containerd/containerd/main/containerd.service
$ mv containerd.service /lib/systemd/system/containerd.service
$ systemctl daemon-reload
$ systemctl enable --now containerd

$ mkdir -p /etc/containerd
$ containerd config default > /etc/containerd/config.toml
$ vim /etc/containerd/config.toml

Change SystemdCgroup to true

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
            BinaryName = ""
            .....
            ShimCgroup = ""
            SystemdCgroup = true

$ systemctl restart containerd

Install runc on all nodes

$ wget https://github.com/opencontainers/runc/releases/download/v1.1.4/runc.amd64
$ install -m 755 runc.amd64 /usr/local/sbin/runc

Install CNI(Container Network Interface) plugin on all nodes

$ wget https://github.com/containernetworking/plugins/releases/download/v1.2.0/cni-plugins-linux-amd64-v1.2.0.tgz
$ mkdir -p /opt/cni/bin
$ tar Cxzvf /opt/cni/bin cni-plugins-linux-amd64-v1.2.0.tgz

Disable apparmor on all nodes

$ systemctl stop apparmor
$ systemctl disable apparmor
$ systemctl restart containerd.service
$ systemctl restart kubelet

Install Nerdctl to all nodes

$ wget https://github.com/containerd/nerdctl/releases/download/v1.2.1/nerdctl-1.2.1-linux-amd64.tar.gz
$ tar Cxzvvf /usr/local/bin nerdctl-1.2.1-linux-amd64.tar.gz

Install Crictl to all nodes

$ wget https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.26.0/crictl-v1.26.0-linux-amd64.tar.gz
$ tar zxvf crictl-v1.26.0-linux-amd64.tar.gz -C /usr/local/bin

Install kubeadm/kubelet to all nodes

$ curl -L --remote-name-all https://dl.k8s.io/release/v1.26.2/bin/linux/amd64/{kubeadm,kubelet}
$ chmod +x {kubeadm,kubelet}
$ mv {kubeadm,kubelet} /usr/local/bin

$ curl -sSL "https://raw.githubusercontent.com/kubernetes/release/v0.4.0/cmd/kubepkg/templates/latest/deb/kubelet/lib/systemd/system/kubelet.service" | sed "s:/usr/bin:/usr/local/bin:g" | sudo tee /etc/systemd/system/kubelet.service
$ mkdir -p /etc/systemd/system/kubelet.service.d
$ curl -sSL "https://raw.githubusercontent.com/kubernetes/release/v0.4.0/cmd/kubepkg/templates/latest/deb/kubeadm/10-kubeadm.conf" | sed "s:/usr/bin:/usr/local/bin:g" | sudo tee /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

$ systemctl enable --now kubelet

Install kubectl to all nodes

$ curl -LO https://dl.k8s.io/release/v1.26.0/bin/linux/amd64/kubectl
$ install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

Configure a cgroup driver and initialise Kubernetes on the master node

$ cat <<EOF > kubeadm-config.yaml
kind: ClusterConfiguration
apiVersion: kubeadm.k8s.io/v1beta3
kubernetesVersion: v1.26.2
---
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
cgroupDriver: systemd
EOF

$ kubeadm init --config kubeadm-config.yaml --upload-certs

• If something happens, then just reset and try again $ kubeadm reset

Copy kubeconfig on the master node

$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

$ kubectl get nodes
$ kubectl get pods --all-namespaces

Install Cilium network on the master node

$ curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/v0.13.0/cilium-linux-amd64.tar.gz
$ tar xzvfC cilium-linux-amd64.tar.gz /usr/local/bin
$ cilium install
$ cilium status --wait
$ kubectl get pods --all-namespaces

Copy ~/.kube/config from the master node to all nodes

Join all other nodes as control plane

$ kubeadm join 192.168.1.150:6443 --token qqe8qp.4enkngnf1qk7a7di \
	--discovery-token-ca-cert-hash sha256:bf750bbcc8f42322a7a9f2d1cc2980f59443d029777760f916693b1d3a753224**

$ kubectl get nodes

Test Cilium connectivity

$ cilium connectivity test

Install Metrics

$ kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml