In the last post we’ve brought up a RKE Kubernetes cluster on a single node. While that is cool for demonstration purposes or testing, this is nothing for a real life setup. Running the control pane, the etcd nodes and the worker nodes all on one node, is nothing you want to do usually, as you can not guarantee fault tolerance with such a setup. To make the RKE cluster highly available we’ll be adding two additional nodes to the configuration in this post. We’ll end up with three nodes, all running etcd, control pane and workers.
Before you can add the additional nodes, they need to be prepared in very much the same way as the first node: Bring the system to the latest release, install a supported version Docker, create the group and the user, and use the same SSH configuration as on the first node.
$ sudo apt update && sudo apt dist-upgrade -y && sudo systemctl reboot $ sudo hostnamectl set-hostname rancher2 $ # sudo hostnamectl set-hostname rancher3 # on the third node $ echo "10.0.1.168 rancher2 rancher2.it.dbi-services.com" >> /etc/hosts $ # echo "10.0.1.168 rancher3 rancher3.it.dbi-services.com" >> /etc/hosts # on the third node $ exit $ sudo curl https://releases.rancher.com/install-docker/19.03.sh | sh $ sudo bash $ echo "rancher ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers $ exit $ sudo systemctl reboot $ sudo groupadd rancher $ sudo useradd -g rancher -G docker -m -s /bin/bash rancher $ sudo passwd rancher $ sudo su - rancher
Before proceeding make sure that you use the same ssk key for the rancher user on the additional nodes, and that you can login from the first node without being prompted for a password:
$ mkdir .ssh $ chmod 700 .ssh/ $ echo "-----BEGIN OPENSSH PRIVATE KEY----- > b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAABFwAAAAdzc2gtcn > NhAAAAAwEAAQAAAQEAx+iJ2W/nGWytnVxyEeRuUDf8UyX3XOxEv7w+TeNGm3o6votXzsEY > CclNxZ0KBt72OnPlpCjNgMOZhKC7XIDwEkhldLyMUVV8jdh/03qfJDyVBp4zqpQ2s1yf/b > SU8cqOrj0gSYmozQdbGybZHmzgj+q9HS5iCAJZ7DUeM43E6kUvHpBJ6a1uP2fIr6+BRd25 > sejcT7kgu50Dv/cVxQ1s0hVydX29kAe0S9IFZUWIlsPCNzPxUGNJxigoC2tAcsXttyeguQ > dtCzTYPgm3wBOoIOR9pAns8kHfiaajZK36vdF6/nEuaI2pw0IpkAct6aFqWq54utgdG9zv > a8mqci/94QAAA8i6pMTbuqTE2wAAAAdzc2gtcnNhAAABAQDH6InZb+cZbK2dXHIR5G5QN/ > xTJfdc7ES/vD5N40abejq+i1fOwRgJyU3FnQoG3vY6c+WkKM2Aw5mEoLtcgPASSGV0vIxR > VXyN2H/Tep8kPJUGnjOqlDazXJ/9tJTxyo6uPSBJiajNB1sbJtkebOCP6r0dLmIIAlnsNR > 4zjcTqRS8ekEnprW4/Z8ivr4FF3bmx6NxPuSC7nQO/9xXFDWzSFXJ1fb2QB7RL0gVlRYiW > w8I3M/FQY0nGKCgLa0Byxe23J6C5B20LNNg+CbfAE6gg5H2kCezyQd+JpqNkrfq90Xr+cS > 5ojanDQimQBy3poWparni62B0b3O9ryapyL/3hAAAAAwEAAQAAAQAueLVK8cOUWnpFoY72 > 79ZhGZKztZi6ZkZZGCaXrqTkUdbEItpnuuWeqMhGjwocrMoqrnSM49tZ+p5+gWrsxyCH74 > J+T7KC2c+ZneGhRNkn8Flob3BtUAUjTv32WXtidgcTJCyUS8cM2o/oUPCaLQ9LBXOvC/BI > ElvbGEIMFAHZv4+eVcZt1NJG3qlu8CXfxRAe6UPLAJOATRyFoNBycPyYu9Hhpr2vXvzksc > QJUT177q2nu5U+UbCAatekQSGVqv18RWnECKJP4ntSbUMhg/PoPQALnWC09epD+397Yqwp > uevR76u7S78q0SnycCvT9EMwpGRjl1e/FTZFejEs9rY9AAAAgQDlMVjYrJ4l5jIrT6GBPE > 7cBBlMW7P0sr1qFxjQQ05JC4CpgCkvqQDqL4alErQ5KTwk9ZsgJY1N49tQk6Rtxv98BK8K > x3d0dth/2q690iDG6LzExTFI26fjPK0a22FLouXSexoQtsHqnpefR9HuJWHPAIhBlgjX98 > Ce/A9McrIfOAAAAIEA/jhYGQaiqhZJIo7ggXVT3yj2ysXjPQ9TR+WRb+Ze3esi/bAUfKfK > 2XtZTALNTFw6+KlorHK5ZgvMdpPLSeAg0htO5g6dLhmVv8VuAItVFQMm/R6AGFc/+EJw9k > iWaGakJzmzCBRwfyZFh3MeMM9sxq60HyV1VHx/SzQvwKNVOJsAAACBAMlO2QU4r1H8kyzu > jn5/NgX0lO6iHDhQWKQywrQ3NjYmtYRBhwpT62MpnpHpev6OpkR2xPOJ+9fDG2K1Q3raSP > jfKaurZlMqmvVeziIhQEXrB3L3vnyq5Jx85oqHv7sh7PYCBD4J6zgL5o66fZOoqdc57GLC > K+XnWjDZpULuQxUzAAAAD3JhbmNoZXJAcmFuZ2VyMQECAw== > -----END OPENSSH PRIVATE KEY-----" > .ssh/id_rsa $ chmod 600 .ssh/id_rsa $ echo "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDH6InZb+cZbK2dXHIR5G5QN/xTJfdc7ES/vD5N40abejq+i1fOwRgJyU3FnQoG3vY6c+WkKM2Aw5mEoLtcgPASSGV0vIxRVXyN2H/Tep8kPJUGnjOqlDazXJ/9tJTxyo6uPSBJiajNB1sbJtkebOCP6r0dLmIIAlnsNR4zjcTqRS8ekEnprW4/Z8ivr4FF3bmx6NxPuSC7nQO/9xXFDWzSFXJ1fb2QB7RL0gVlRYiWw8I3M/FQY0nGKCgLa0Byxe23J6C5B20LNNg+CbfAE6gg5H2kCezyQd+JpqNkrfq90Xr+cS5ojanDQimQBy3poWparni62B0b3O9ryapyL/3h rancher@rancher1" >> .ssh/authorized_keys rancher@rancher1:~$ ssh 10.0.1.253 The authenticity of host '10.0.1.253 (10.0.1.253)' can't be established. ECDSA key fingerprint is SHA256:/JzK5lFQv6qsM5zi4A+1JYwS5u0Iup3uUUV8927MF50. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '10.0.1.253' (ECDSA) to the list of known hosts. Linux rancher2 4.19.0-14-cloud-amd64 #1 SMP Debian 4.19.171-2 (2021-01-30) x86_64 The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. rancher@rancher2:~$ logout Connection to 10.0.1.253 closed. rancher@rancher1:~$ ssh 10.0.1.73 The authenticity of host '10.0.1.73 (10.0.1.73)' can't be established. ECDSA key fingerprint is SHA256:oVfRCbqh5PIdTx16+wNmMS8CNnHTnQXsjlpybHmPVlY. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '10.0.1.73' (ECDSA) to the list of known hosts. Linux rancher3 4.19.0-14-cloud-amd64 #1 SMP Debian 4.19.171-2 (2021-01-30) x86_64 The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law.
Once that is confirmed, we need to adjust the RKE cluster configuration file, to include the new nodes. Currently the node section looks like this:
# please consult the documentation on how to configure custom RKE images. nodes: - address: 10.0.1.168 port: "22" internal_address: "" role: - controlplane - worker - etcd hostname_override: "" user: rancher docker_socket: /var/run/docker.sock ssh_key: "" ssh_key_path: ~/.ssh/id_rsa ssh_cert: "" ssh_cert_path: "" labels: {} taints: []
We need to add the two additional nodes. As this setup is on EC2, you need to specify the public and the internal IP addresses:
The node section in the yaml file looks like this (I am assuming that you are familiar with security groups and traffic is allowed between the node):
nodes: - address: 18.195.249.125 port: "22" internal_address: "10.0.1.168" role: - controlplane - worker - etcd hostname_override: "rancher1" user: rancher docker_socket: /var/run/docker.sock ssh_key: "" ssh_key_path: ~/.ssh/id_rsa ssh_cert: "" ssh_cert_path: "" labels: {} taints: [] - address: 3.64.193.173 port: "22" internal_address: "10.0.1.253" role: - controlplane - worker - etcd hostname_override: "rancher2" user: rancher docker_socket: /var/run/docker.sock ssh_key: "" ssh_key_path: ~/.ssh/id_rsa ssh_cert: "" ssh_cert_path: "" labels: {} taints: [] - address: 18.185.105.131 port: "22" internal_address: "10.0.1.73" role: - controlplane - worker - etcd hostname_override: "rancher3" user: rancher docker_socket: /var/run/docker.sock ssh_key: "" ssh_key_path: ~/.ssh/id_rsa ssh_cert: "" ssh_cert_path: "" labels: {} taints: []
That’s all you need to do. Use “rke up” to apply the changed configuration:
rancher@rancher1:~$ rke up INFO[0000] Running RKE version: v1.1.15 INFO[0000] Initiating Kubernetes cluster INFO[0000] [dialer] Setup tunnel for host [3.64.193.173] INFO[0000] [dialer] Setup tunnel for host [18.185.105.131] INFO[0000] [dialer] Setup tunnel for host [18.195.249.125] INFO[0000] Checking if container [cluster-state-deployer] is running on host [3.64.193.173], try #1 INFO[0000] Image [rancher/rke-tools:v0.1.72] exists on host [3.64.193.173] INFO[0000] Starting container [cluster-state-deployer] on host [3.64.193.173], try #1 INFO[0000] [state] Successfully started [cluster-state-deployer] container on host [3.64.193.173] INFO[0000] Checking if container [cluster-state-deployer] is running on host [18.185.105.131], try #1 INFO[0000] Image [rancher/rke-tools:v0.1.72] exists on host [18.185.105.131] INFO[0000] Starting container [cluster-state-deployer] on host [18.185.105.131], try #1 INFO[0001] [state] Successfully started [cluster-state-deployer] container on host [18.185.105.131] INFO[0001] Checking if container [cluster-state-deployer] is running on host [18.195.249.125], try #1 INFO[0001] [certificates] Generating CA kubernetes certificates INFO[0001] [certificates] Generating Kubernetes API server aggregation layer requestheader client CA certificates INFO[0002] [certificates] GenerateServingCertificate is disabled, checking if there are unused kubelet certificates INFO[0002] [certificates] Generating Kubernetes API server certificates INFO[0003] [certificates] Generating Service account token key INFO[0003] [certificates] Generating Kube Controller certificates INFO[0003] [certificates] Generating Kube Scheduler certificates INFO[0003] [certificates] Generating Kube Proxy certificates INFO[0003] [certificates] Generating Node certificate INFO[0003] [certificates] Generating admin certificates and kubeconfig INFO[0003] [certificates] Generating Kubernetes API server proxy client certificates INFO[0004] [certificates] Generating kube-etcd-10-0-1-168 certificate and key INFO[0004] [certificates] Generating kube-etcd-10-0-1-253 certificate and key INFO[0004] [certificates] Generating kube-etcd-10-0-1-73 certificate and key INFO[0005] Successfully Deployed state file at [./cluster.rkestate] INFO[0005] Building Kubernetes cluster INFO[0005] [dialer] Setup tunnel for host [18.185.105.131] INFO[0005] [dialer] Setup tunnel for host [18.195.249.125] INFO[0005] [dialer] Setup tunnel for host [3.64.193.173] INFO[0005] [network] Deploying port listener containers INFO[0005] Image [rancher/rke-tools:v0.1.72] exists on host [3.64.193.173] INFO[0005] Image [rancher/rke-tools:v0.1.72] exists on host [18.185.105.131] INFO[0005] Image [rancher/rke-tools:v0.1.72] exists on host [18.195.249.125] INFO[0005] Starting container [rke-etcd-port-listener] on host [18.185.105.131], try #1 INFO[0005] Starting container [rke-etcd-port-listener] on host [18.195.249.125], try #1 INFO[0005] Starting container [rke-etcd-port-listener] on host [3.64.193.173], try #1 INFO[0005] [network] Successfully started [rke-etcd-port-listener] container on host [18.185.105.131] INFO[0005] [network] Successfully started [rke-etcd-port-listener] container on host [18.195.249.125] INFO[0005] [network] Successfully started [rke-etcd-port-listener] container on host [3.64.193.173] INFO[0005] Image [rancher/rke-tools:v0.1.72] exists on host [3.64.193.173] INFO[0005] Image [rancher/rke-tools:v0.1.72] exists on host [18.195.249.125] INFO[0005] Image [rancher/rke-tools:v0.1.72] exists on host [18.185.105.131] INFO[0006] Starting container [rke-cp-port-listener] on host [18.195.249.125], try #1 INFO[0006] Starting container [rke-cp-port-listener] on host [18.185.105.131], try #1 INFO[0006] Starting container [rke-cp-port-listener] on host [3.64.193.173], try #1 INFO[0006] [network] Successfully started [rke-cp-port-listener] container on host [3.64.193.173] INFO[0006] [network] Successfully started [rke-cp-port-listener] container on host [18.185.105.131] INFO[0006] [network] Successfully started [rke-cp-port-listener] container on host [18.195.249.125] INFO[0006] Image [rancher/rke-tools:v0.1.72] exists on host [18.195.249.125] INFO[0006] Image [rancher/rke-tools:v0.1.72] exists on host [3.64.193.173] INFO[0006] Image [rancher/rke-tools:v0.1.72] exists on host [18.185.105.131] INFO[0006] Starting container [rke-worker-port-listener] on host [18.185.105.131], try #1 INFO[0006] Starting container [rke-worker-port-listener] on host [18.195.249.125], try #1 INFO[0006] Starting container [rke-worker-port-listener] on host [3.64.193.173], try #1 INFO[0006] [network] Successfully started [rke-worker-port-listener] container on host [3.64.193.173] INFO[0006] [network] Successfully started [rke-worker-port-listener] container on host [18.185.105.131] INFO[0006] [network] Successfully started [rke-worker-port-listener] container on host [18.195.249.125] INFO[0006] [network] Port listener containers deployed successfully INFO[0006] [network] Running etcd etcd port checks INFO[0006] Image [rancher/rke-tools:v0.1.72] exists on host [3.64.193.173] INFO[0006] Image [rancher/rke-tools:v0.1.72] exists on host [18.195.249.125] INFO[0006] Image [rancher/rke-tools:v0.1.72] exists on host [18.185.105.131] INFO[0007] Starting container [rke-port-checker] on host [18.185.105.131], try #1 INFO[0007] Starting container [rke-port-checker] on host [3.64.193.173], try #1 INFO[0007] Starting container [rke-port-checker] on host [18.195.249.125], try #1 INFO[0007] [network] Successfully started [rke-port-checker] container on host [3.64.193.173] INFO[0007] [network] Successfully started [rke-port-checker] container on host [18.185.105.131] INFO[0007] [network] Successfully started [rke-port-checker] container on host [18.195.249.125] INFO[0007] Removing container [rke-port-checker] on host [3.64.193.173], try #1 INFO[0007] Removing container [rke-port-checker] on host [18.195.249.125], try #1 INFO[0008] Removing container [rke-port-checker] on host [18.185.105.131], try #1 INFO[0008] [network] Running control plane -> etcd port checks INFO[0008] Image [rancher/rke-tools:v0.1.72] exists on host [18.185.105.131] INFO[0008] Image [rancher/rke-tools:v0.1.72] exists on host [18.195.249.125] INFO[0008] Image [rancher/rke-tools:v0.1.72] exists on host [3.64.193.173] INFO[0008] Starting container [rke-port-checker] on host [18.195.249.125], try #1 INFO[0008] Starting container [rke-port-checker] on host [3.64.193.173], try #1 INFO[0008] Starting container [rke-port-checker] on host [18.185.105.131], try #1 INFO[0008] [network] Successfully started [rke-port-checker] container on host [18.195.249.125] INFO[0008] [network] Successfully started [rke-port-checker] container on host [3.64.193.173] INFO[0008] [network] Successfully started [rke-port-checker] container on host [18.185.105.131] INFO[0008] Removing container [rke-port-checker] on host [18.195.249.125], try #1 INFO[0008] Removing container [rke-port-checker] on host [3.64.193.173], try #1 INFO[0008] Removing container [rke-port-checker] on host [18.185.105.131], try #1 INFO[0008] [network] Running control plane -> worker port checks INFO[0008] Image [rancher/rke-tools:v0.1.72] exists on host [3.64.193.173] INFO[0008] Image [rancher/rke-tools:v0.1.72] exists on host [18.185.105.131] INFO[0008] Image [rancher/rke-tools:v0.1.72] exists on host [18.195.249.125] INFO[0009] Starting container [rke-port-checker] on host [18.185.105.131], try #1 INFO[0009] Starting container [rke-port-checker] on host [18.195.249.125], try #1 INFO[0009] Starting container [rke-port-checker] on host [3.64.193.173], try #1 INFO[0009] [network] Successfully started [rke-port-checker] container on host [18.195.249.125] INFO[0009] [network] Successfully started [rke-port-checker] container on host [18.185.105.131] INFO[0009] [network] Successfully started [rke-port-checker] container on host [3.64.193.173] INFO[0009] Removing container [rke-port-checker] on host [18.195.249.125], try #1 INFO[0009] Removing container [rke-port-checker] on host [18.185.105.131], try #1 INFO[0009] Removing container [rke-port-checker] on host [3.64.193.173], try #1 INFO[0009] [network] Running workers -> control plane port checks INFO[0009] Image [rancher/rke-tools:v0.1.72] exists on host [3.64.193.173] INFO[0009] Image [rancher/rke-tools:v0.1.72] exists on host [18.195.249.125] INFO[0009] Image [rancher/rke-tools:v0.1.72] exists on host [18.185.105.131] INFO[0009] Starting container [rke-port-checker] on host [3.64.193.173], try #1 INFO[0009] Starting container [rke-port-checker] on host [18.185.105.131], try #1 INFO[0009] Starting container [rke-port-checker] on host [18.195.249.125], try #1 INFO[0009] [network] Successfully started [rke-port-checker] container on host [3.64.193.173] INFO[0009] [network] Successfully started [rke-port-checker] container on host [18.185.105.131] INFO[0009] [network] Successfully started [rke-port-checker] container on host [18.195.249.125] INFO[0009] Removing container [rke-port-checker] on host [3.64.193.173], try #1 INFO[0009] Removing container [rke-port-checker] on host [18.185.105.131], try #1 INFO[0009] Removing container [rke-port-checker] on host [18.195.249.125], try #1 INFO[0009] [network] Checking KubeAPI port Control Plane hosts INFO[0009] [network] Removing port listener containers INFO[0009] Removing container [rke-etcd-port-listener] on host [18.195.249.125], try #1 INFO[0009] Removing container [rke-etcd-port-listener] on host [18.185.105.131], try #1 INFO[0009] Removing container [rke-etcd-port-listener] on host [3.64.193.173], try #1 INFO[0010] [remove/rke-etcd-port-listener] Successfully removed container on host [3.64.193.173] INFO[0010] [remove/rke-etcd-port-listener] Successfully removed container on host [18.185.105.131] INFO[0010] [remove/rke-etcd-port-listener] Successfully removed container on host [18.195.249.125] INFO[0010] Removing container [rke-cp-port-listener] on host [18.195.249.125], try #1 INFO[0010] Removing container [rke-cp-port-listener] on host [3.64.193.173], try #1 INFO[0010] Removing container [rke-cp-port-listener] on host [18.185.105.131], try #1 INFO[0010] [remove/rke-cp-port-listener] Successfully removed container on host [18.185.105.131] INFO[0010] [remove/rke-cp-port-listener] Successfully removed container on host [18.195.249.125] INFO[0010] [remove/rke-cp-port-listener] Successfully removed container on host [3.64.193.173] INFO[0010] Removing container [rke-worker-port-listener] on host [18.195.249.125], try #1 INFO[0010] Removing container [rke-worker-port-listener] on host [18.185.105.131], try #1 INFO[0010] Removing container [rke-worker-port-listener] on host [3.64.193.173], try #1 INFO[0010] [remove/rke-worker-port-listener] Successfully removed container on host [3.64.193.173] INFO[0010] [remove/rke-worker-port-listener] Successfully removed container on host [18.185.105.131] INFO[0010] [remove/rke-worker-port-listener] Successfully removed container on host [18.195.249.125] INFO[0010] [network] Port listener containers removed successfully INFO[0010] [certificates] Deploying kubernetes certificates to Cluster nodes INFO[0010] Checking if container [cert-deployer] is running on host [18.195.249.125], try #1 INFO[0010] Checking if container [cert-deployer] is running on host [3.64.193.173], try #1 INFO[0010] Checking if container [cert-deployer] is running on host [18.185.105.131], try #1 INFO[0010] Image [rancher/rke-tools:v0.1.72] exists on host [18.185.105.131] INFO[0010] Image [rancher/rke-tools:v0.1.72] exists on host [3.64.193.173] INFO[0010] Image [rancher/rke-tools:v0.1.72] exists on host [18.195.249.125] INFO[0011] Starting container [cert-deployer] on host [3.64.193.173], try #1 INFO[0011] Starting container [cert-deployer] on host [18.185.105.131], try #1 INFO[0011] Starting container [cert-deployer] on host [18.195.249.125], try #1 INFO[0011] Checking if container [cert-deployer] is running on host [3.64.193.173], try #1 INFO[0011] Checking if container [cert-deployer] is running on host [18.185.105.131], try #1 INFO[0011] Checking if container [cert-deployer] is running on host [18.195.249.125], try #1 INFO[0016] Checking if container [cert-deployer] is running on host [3.64.193.173], try #1 INFO[0016] Removing container [cert-deployer] on host [3.64.193.173], try #1 INFO[0016] Checking if container [cert-deployer] is running on host [18.185.105.131], try #1 INFO[0016] Removing container [cert-deployer] on host [18.185.105.131], try #1 INFO[0016] Checking if container [cert-deployer] is running on host [18.195.249.125], try #1 INFO[0016] Removing container [cert-deployer] on host [18.195.249.125], try #1 INFO[0016] [reconcile] Rebuilding and updating local kube config INFO[0016] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml] INFO[0016] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml] INFO[0016] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml] INFO[0016] [certificates] Successfully deployed kubernetes certificates to Cluster nodes INFO[0016] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [18.195.249.125] INFO[0016] Image [rancher/rke-tools:v0.1.72] exists on host [18.195.249.125] INFO[0016] Starting container [file-deployer] on host [18.195.249.125], try #1 INFO[0017] Successfully started [file-deployer] container on host [18.195.249.125] INFO[0017] Waiting for [file-deployer] container to exit on host [18.195.249.125] INFO[0017] Waiting for [file-deployer] container to exit on host [18.195.249.125] INFO[0017] Container [file-deployer] is still running on host [18.195.249.125]: stderr: [], stdout: [] INFO[0018] Waiting for [file-deployer] container to exit on host [18.195.249.125] INFO[0018] Removing container [file-deployer] on host [18.195.249.125], try #1 INFO[0018] [remove/file-deployer] Successfully removed container on host [18.195.249.125] INFO[0018] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [3.64.193.173] INFO[0018] Image [rancher/rke-tools:v0.1.72] exists on host [3.64.193.173] INFO[0018] Starting container [file-deployer] on host [3.64.193.173], try #1 INFO[0018] Successfully started [file-deployer] container on host [3.64.193.173] INFO[0018] Waiting for [file-deployer] container to exit on host [3.64.193.173] INFO[0018] Waiting for [file-deployer] container to exit on host [3.64.193.173] INFO[0018] Container [file-deployer] is still running on host [3.64.193.173]: stderr: [], stdout: [] INFO[0019] Waiting for [file-deployer] container to exit on host [3.64.193.173] INFO[0019] Removing container [file-deployer] on host [3.64.193.173], try #1 INFO[0019] [remove/file-deployer] Successfully removed container on host [3.64.193.173] INFO[0019] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [18.185.105.131] INFO[0019] Image [rancher/rke-tools:v0.1.72] exists on host [18.185.105.131] INFO[0019] Starting container [file-deployer] on host [18.185.105.131], try #1 INFO[0020] Successfully started [file-deployer] container on host [18.185.105.131] INFO[0020] Waiting for [file-deployer] container to exit on host [18.185.105.131] INFO[0020] Waiting for [file-deployer] container to exit on host [18.185.105.131] INFO[0020] Container [file-deployer] is still running on host [18.185.105.131]: stderr: [], stdout: [] INFO[0021] Waiting for [file-deployer] container to exit on host [18.185.105.131] INFO[0021] Removing container [file-deployer] on host [18.185.105.131], try #1 INFO[0021] [remove/file-deployer] Successfully removed container on host [18.185.105.131] INFO[0021] [/etc/kubernetes/audit-policy.yaml] Successfully deployed audit policy file to Cluster control nodes INFO[0021] [reconcile] Reconciling cluster state INFO[0021] [reconcile] This is newly generated cluster INFO[0021] Pre-pulling kubernetes images INFO[0021] Pulling image [rancher/hyperkube:v1.18.16-rancher1] on host [18.185.105.131], try #1 INFO[0021] Pulling image [rancher/hyperkube:v1.18.16-rancher1] on host [3.64.193.173], try #1 INFO[0021] Image [rancher/hyperkube:v1.18.16-rancher1] exists on host [18.195.249.125] INFO[0047] Image [rancher/hyperkube:v1.18.16-rancher1] exists on host [3.64.193.173] INFO[0047] Image [rancher/hyperkube:v1.18.16-rancher1] exists on host [18.185.105.131] INFO[0047] Kubernetes images pulled successfully INFO[0047] [etcd] Building up etcd plane.. INFO[0047] Image [rancher/rke-tools:v0.1.72] exists on host [18.195.249.125] INFO[0047] Starting container [etcd-fix-perm] on host [18.195.249.125], try #1 INFO[0047] Successfully started [etcd-fix-perm] container on host [18.195.249.125] INFO[0047] Waiting for [etcd-fix-perm] container to exit on host [18.195.249.125] INFO[0047] Waiting for [etcd-fix-perm] container to exit on host [18.195.249.125] INFO[0047] Container [etcd-fix-perm] is still running on host [18.195.249.125]: stderr: [], stdout: [] INFO[0048] Waiting for [etcd-fix-perm] container to exit on host [18.195.249.125] INFO[0048] Removing container [etcd-fix-perm] on host [18.195.249.125], try #1 INFO[0048] [remove/etcd-fix-perm] Successfully removed container on host [18.195.249.125] INFO[0048] Image [rancher/coreos-etcd:v3.4.3-rancher1] exists on host [18.195.249.125] INFO[0048] Starting container [etcd] on host [18.195.249.125], try #1 INFO[0049] [etcd] Successfully started [etcd] container on host [18.195.249.125] INFO[0049] [etcd] Running rolling snapshot container [etcd-snapshot-once] on host [18.195.249.125] INFO[0049] Image [rancher/rke-tools:v0.1.72] exists on host [18.195.249.125] INFO[0049] Starting container [etcd-rolling-snapshots] on host [18.195.249.125], try #1 INFO[0049] [etcd] Successfully started [etcd-rolling-snapshots] container on host [18.195.249.125] INFO[0054] Image [rancher/rke-tools:v0.1.72] exists on host [18.195.249.125] INFO[0054] Starting container [rke-bundle-cert] on host [18.195.249.125], try #1 INFO[0054] [certificates] Successfully started [rke-bundle-cert] container on host [18.195.249.125] INFO[0054] Waiting for [rke-bundle-cert] container to exit on host [18.195.249.125] INFO[0054] Container [rke-bundle-cert] is still running on host [18.195.249.125]: stderr: [], stdout: [] INFO[0055] Waiting for [rke-bundle-cert] container to exit on host [18.195.249.125] INFO[0055] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [18.195.249.125] INFO[0055] Removing container [rke-bundle-cert] on host [18.195.249.125], try #1 INFO[0056] Image [rancher/rke-tools:v0.1.72] exists on host [18.195.249.125] INFO[0056] Starting container [rke-log-linker] on host [18.195.249.125], try #1 INFO[0056] [etcd] Successfully started [rke-log-linker] container on host [18.195.249.125] INFO[0056] Removing container [rke-log-linker] on host [18.195.249.125], try #1 INFO[0056] [remove/rke-log-linker] Successfully removed container on host [18.195.249.125] INFO[0056] Image [rancher/rke-tools:v0.1.72] exists on host [3.64.193.173] INFO[0062] Starting container [etcd-fix-perm] on host [3.64.193.173], try #1 INFO[0062] Successfully started [etcd-fix-perm] container on host [3.64.193.173] INFO[0062] Waiting for [etcd-fix-perm] container to exit on host [3.64.193.173] INFO[0062] Waiting for [etcd-fix-perm] container to exit on host [3.64.193.173] INFO[0062] Container [etcd-fix-perm] is still running on host [3.64.193.173]: stderr: [], stdout: [] INFO[0063] Waiting for [etcd-fix-perm] container to exit on host [3.64.193.173] INFO[0063] Removing container [etcd-fix-perm] on host [3.64.193.173], try #1 INFO[0063] [remove/etcd-fix-perm] Successfully removed container on host [3.64.193.173] INFO[0063] Pulling image [rancher/coreos-etcd:v3.4.3-rancher1] on host [3.64.193.173], try #1 INFO[0067] Image [rancher/coreos-etcd:v3.4.3-rancher1] exists on host [3.64.193.173] INFO[0067] Starting container [etcd] on host [3.64.193.173], try #1 INFO[0067] [etcd] Successfully started [etcd] container on host [3.64.193.173] INFO[0067] [etcd] Running rolling snapshot container [etcd-snapshot-once] on host [3.64.193.173] INFO[0067] Image [rancher/rke-tools:v0.1.72] exists on host [3.64.193.173] INFO[0067] Starting container [etcd-rolling-snapshots] on host [3.64.193.173], try #1 INFO[0067] [etcd] Successfully started [etcd-rolling-snapshots] container on host [3.64.193.173] INFO[0072] Image [rancher/rke-tools:v0.1.72] exists on host [3.64.193.173] INFO[0073] Starting container [rke-bundle-cert] on host [3.64.193.173], try #1 INFO[0073] [certificates] Successfully started [rke-bundle-cert] container on host [3.64.193.173] INFO[0073] Waiting for [rke-bundle-cert] container to exit on host [3.64.193.173] INFO[0073] Container [rke-bundle-cert] is still running on host [3.64.193.173]: stderr: [], stdout: [] INFO[0074] Waiting for [rke-bundle-cert] container to exit on host [3.64.193.173] INFO[0074] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [3.64.193.173] INFO[0074] Removing container [rke-bundle-cert] on host [3.64.193.173], try #1 INFO[0074] Image [rancher/rke-tools:v0.1.72] exists on host [3.64.193.173] INFO[0074] Starting container [rke-log-linker] on host [3.64.193.173], try #1 INFO[0074] [etcd] Successfully started [rke-log-linker] container on host [3.64.193.173] INFO[0074] Removing container [rke-log-linker] on host [3.64.193.173], try #1 INFO[0075] [remove/rke-log-linker] Successfully removed container on host [3.64.193.173] INFO[0075] Image [rancher/rke-tools:v0.1.72] exists on host [18.185.105.131] INFO[0078] Starting container [etcd-fix-perm] on host [18.185.105.131], try #1 INFO[0079] Successfully started [etcd-fix-perm] container on host [18.185.105.131] INFO[0079] Waiting for [etcd-fix-perm] container to exit on host [18.185.105.131] INFO[0079] Waiting for [etcd-fix-perm] container to exit on host [18.185.105.131] INFO[0079] Container [etcd-fix-perm] is still running on host [18.185.105.131]: stderr: [], stdout: [] INFO[0080] Waiting for [etcd-fix-perm] container to exit on host [18.185.105.131] INFO[0080] Removing container [etcd-fix-perm] on host [18.185.105.131], try #1 INFO[0080] [remove/etcd-fix-perm] Successfully removed container on host [18.185.105.131] INFO[0080] Pulling image [rancher/coreos-etcd:v3.4.3-rancher1] on host [18.185.105.131], try #1 INFO[0084] Image [rancher/coreos-etcd:v3.4.3-rancher1] exists on host [18.185.105.131] INFO[0084] Starting container [etcd] on host [18.185.105.131], try #1 INFO[0084] [etcd] Successfully started [etcd] container on host [18.185.105.131] INFO[0084] [etcd] Running rolling snapshot container [etcd-snapshot-once] on host [18.185.105.131] INFO[0084] Image [rancher/rke-tools:v0.1.72] exists on host [18.185.105.131] INFO[0084] Starting container [etcd-rolling-snapshots] on host [18.185.105.131], try #1 INFO[0084] [etcd] Successfully started [etcd-rolling-snapshots] container on host [18.185.105.131] INFO[0089] Image [rancher/rke-tools:v0.1.72] exists on host [18.185.105.131] INFO[0089] Starting container [rke-bundle-cert] on host [18.185.105.131], try #1 INFO[0090] [certificates] Successfully started [rke-bundle-cert] container on host [18.185.105.131] INFO[0090] Waiting for [rke-bundle-cert] container to exit on host [18.185.105.131] INFO[0090] Container [rke-bundle-cert] is still running on host [18.185.105.131]: stderr: [], stdout: [] INFO[0091] Waiting for [rke-bundle-cert] container to exit on host [18.185.105.131] INFO[0091] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [18.185.105.131] INFO[0091] Removing container [rke-bundle-cert] on host [18.185.105.131], try #1 INFO[0091] Image [rancher/rke-tools:v0.1.72] exists on host [18.185.105.131] INFO[0091] Starting container [rke-log-linker] on host [18.185.105.131], try #1 INFO[0091] [etcd] Successfully started [rke-log-linker] container on host [18.185.105.131] INFO[0091] Removing container [rke-log-linker] on host [18.185.105.131], try #1 INFO[0092] [remove/rke-log-linker] Successfully removed container on host [18.185.105.131] INFO[0092] [etcd] Successfully started etcd plane.. Checking etcd cluster health INFO[0092] [etcd] etcd host [18.195.249.125] reported healthy=true INFO[0092] [controlplane] Building up Controller Plane.. INFO[0092] Checking if container [service-sidekick] is running on host [18.195.249.125], try #1 INFO[0092] Checking if container [service-sidekick] is running on host [18.185.105.131], try #1 INFO[0092] Checking if container [service-sidekick] is running on host [3.64.193.173], try #1 INFO[0092] Image [rancher/rke-tools:v0.1.72] exists on host [3.64.193.173] INFO[0092] Image [rancher/rke-tools:v0.1.72] exists on host [18.185.105.131] INFO[0092] Image [rancher/rke-tools:v0.1.72] exists on host [18.195.249.125] INFO[0092] Image [rancher/hyperkube:v1.18.16-rancher1] exists on host [18.185.105.131] INFO[0092] Image [rancher/hyperkube:v1.18.16-rancher1] exists on host [3.64.193.173] INFO[0092] Image [rancher/hyperkube:v1.18.16-rancher1] exists on host [18.195.249.125] INFO[0092] Starting container [kube-apiserver] on host [18.185.105.131], try #1 INFO[0092] Starting container [kube-apiserver] on host [3.64.193.173], try #1 INFO[0092] Starting container [kube-apiserver] on host [18.195.249.125], try #1 INFO[0092] [controlplane] Successfully started [kube-apiserver] container on host [18.185.105.131] INFO[0092] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [18.185.105.131] INFO[0092] [controlplane] Successfully started [kube-apiserver] container on host [18.195.249.125] INFO[0092] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [18.195.249.125] INFO[0092] [controlplane] Successfully started [kube-apiserver] container on host [3.64.193.173] INFO[0092] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [3.64.193.173] INFO[0102] [healthcheck] service [kube-apiserver] on host [18.185.105.131] is healthy INFO[0102] Image [rancher/rke-tools:v0.1.72] exists on host [18.185.105.131] INFO[0102] [healthcheck] service [kube-apiserver] on host [3.64.193.173] is healthy INFO[0102] [healthcheck] service [kube-apiserver] on host [18.195.249.125] is healthy INFO[0102] Image [rancher/rke-tools:v0.1.72] exists on host [3.64.193.173] INFO[0102] Image [rancher/rke-tools:v0.1.72] exists on host [18.195.249.125] INFO[0102] Starting container [rke-log-linker] on host [18.185.105.131], try #1 INFO[0102] Starting container [rke-log-linker] on host [3.64.193.173], try #1 INFO[0102] Starting container [rke-log-linker] on host [18.195.249.125], try #1 INFO[0103] [controlplane] Successfully started [rke-log-linker] container on host [18.185.105.131] INFO[0103] Removing container [rke-log-linker] on host [18.185.105.131], try #1 INFO[0103] [controlplane] Successfully started [rke-log-linker] container on host [3.64.193.173] INFO[0103] Removing container [rke-log-linker] on host [3.64.193.173], try #1 INFO[0103] [controlplane] Successfully started [rke-log-linker] container on host [18.195.249.125] INFO[0103] Removing container [rke-log-linker] on host [18.195.249.125], try #1 INFO[0103] [remove/rke-log-linker] Successfully removed container on host [18.185.105.131] INFO[0103] Image [rancher/hyperkube:v1.18.16-rancher1] exists on host [18.185.105.131] INFO[0103] [remove/rke-log-linker] Successfully removed container on host [3.64.193.173] INFO[0103] Image [rancher/hyperkube:v1.18.16-rancher1] exists on host [3.64.193.173] INFO[0103] [remove/rke-log-linker] Successfully removed container on host [18.195.249.125] INFO[0103] Image [rancher/hyperkube:v1.18.16-rancher1] exists on host [18.195.249.125] INFO[0103] Starting container [kube-controller-manager] on host [18.185.105.131], try #1 INFO[0103] Starting container [kube-controller-manager] on host [3.64.193.173], try #1 INFO[0103] Starting container [kube-controller-manager] on host [18.195.249.125], try #1 INFO[0103] [controlplane] Successfully started [kube-controller-manager] container on host [18.185.105.131] INFO[0103] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [18.185.105.131] INFO[0103] [controlplane] Successfully started [kube-controller-manager] container on host [3.64.193.173] INFO[0103] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [3.64.193.173] INFO[0103] [controlplane] Successfully started [kube-controller-manager] container on host [18.195.249.125] INFO[0103] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [18.195.249.125] INFO[0108] [healthcheck] service [kube-controller-manager] on host [18.185.105.131] is healthy INFO[0108] Image [rancher/rke-tools:v0.1.72] exists on host [18.185.105.131] INFO[0108] [healthcheck] service [kube-controller-manager] on host [3.64.193.173] is healthy INFO[0108] Image [rancher/rke-tools:v0.1.72] exists on host [3.64.193.173] INFO[0108] [healthcheck] service [kube-controller-manager] on host [18.195.249.125] is healthy INFO[0108] Image [rancher/rke-tools:v0.1.72] exists on host [18.195.249.125] INFO[0109] Starting container [rke-log-linker] on host [18.185.105.131], try #1 INFO[0109] Starting container [rke-log-linker] on host [3.64.193.173], try #1 INFO[0109] Starting container [rke-log-linker] on host [18.195.249.125], try #1 INFO[0109] [controlplane] Successfully started [rke-log-linker] container on host [18.185.105.131] INFO[0109] [controlplane] Successfully started [rke-log-linker] container on host [3.64.193.173] INFO[0109] Removing container [rke-log-linker] on host [3.64.193.173], try #1 INFO[0109] Removing container [rke-log-linker] on host [18.185.105.131], try #1 INFO[0109] [controlplane] Successfully started [rke-log-linker] container on host [18.195.249.125] INFO[0109] Removing container [rke-log-linker] on host [18.195.249.125], try #1 INFO[0109] [remove/rke-log-linker] Successfully removed container on host [3.64.193.173] INFO[0109] Image [rancher/hyperkube:v1.18.16-rancher1] exists on host [3.64.193.173] INFO[0109] [remove/rke-log-linker] Successfully removed container on host [18.185.105.131] INFO[0109] Image [rancher/hyperkube:v1.18.16-rancher1] exists on host [18.185.105.131] INFO[0109] Starting container [kube-scheduler] on host [3.64.193.173], try #1 INFO[0109] [remove/rke-log-linker] Successfully removed container on host [18.195.249.125] INFO[0109] Image [rancher/hyperkube:v1.18.16-rancher1] exists on host [18.195.249.125] INFO[0109] Starting container [kube-scheduler] on host [18.185.105.131], try #1 INFO[0109] Starting container [kube-scheduler] on host [18.195.249.125], try #1 INFO[0109] [controlplane] Successfully started [kube-scheduler] container on host [3.64.193.173] INFO[0109] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [3.64.193.173] INFO[0109] [controlplane] Successfully started [kube-scheduler] container on host [18.185.105.131] INFO[0109] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [18.185.105.131] INFO[0109] [controlplane] Successfully started [kube-scheduler] container on host [18.195.249.125] INFO[0109] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [18.195.249.125] INFO[0115] [healthcheck] service [kube-scheduler] on host [3.64.193.173] is healthy INFO[0115] [healthcheck] service [kube-scheduler] on host [18.185.105.131] is healthy INFO[0115] Image [rancher/rke-tools:v0.1.72] exists on host [3.64.193.173] INFO[0115] Image [rancher/rke-tools:v0.1.72] exists on host [18.185.105.131] INFO[0115] [healthcheck] service [kube-scheduler] on host [18.195.249.125] is healthy INFO[0115] Image [rancher/rke-tools:v0.1.72] exists on host [18.195.249.125] INFO[0115] Starting container [rke-log-linker] on host [3.64.193.173], try #1 INFO[0115] Starting container [rke-log-linker] on host [18.185.105.131], try #1 INFO[0115] Starting container [rke-log-linker] on host [18.195.249.125], try #1 INFO[0115] [controlplane] Successfully started [rke-log-linker] container on host [3.64.193.173] INFO[0115] Removing container [rke-log-linker] on host [3.64.193.173], try #1 INFO[0115] [controlplane] Successfully started [rke-log-linker] container on host [18.185.105.131] INFO[0115] Removing container [rke-log-linker] on host [18.185.105.131], try #1 INFO[0115] [controlplane] Successfully started [rke-log-linker] container on host [18.195.249.125] INFO[0115] Removing container [rke-log-linker] on host [18.195.249.125], try #1 INFO[0115] [remove/rke-log-linker] Successfully removed container on host [3.64.193.173] INFO[0115] [remove/rke-log-linker] Successfully removed container on host [18.185.105.131] INFO[0116] [remove/rke-log-linker] Successfully removed container on host [18.195.249.125] INFO[0116] [controlplane] Successfully started Controller Plane.. INFO[0116] [authz] Creating rke-job-deployer ServiceAccount INFO[0116] [authz] rke-job-deployer ServiceAccount created successfully INFO[0116] [authz] Creating system:node ClusterRoleBinding INFO[0116] [authz] system:node ClusterRoleBinding created successfully INFO[0116] [authz] Creating kube-apiserver proxy ClusterRole and ClusterRoleBinding INFO[0116] [authz] kube-apiserver proxy ClusterRole and ClusterRoleBinding created successfully INFO[0116] Successfully Deployed state file at [./cluster.rkestate] INFO[0116] [state] Saving full cluster state to Kubernetes INFO[0116] [state] Successfully Saved full cluster state to Kubernetes ConfigMap: full-cluster-state INFO[0116] [worker] Building up Worker Plane.. INFO[0116] Checking if container [service-sidekick] is running on host [18.185.105.131], try #1 INFO[0116] Checking if container [service-sidekick] is running on host [18.195.249.125], try #1 INFO[0116] Checking if container [service-sidekick] is running on host [3.64.193.173], try #1 INFO[0116] [sidekick] Sidekick container already created on host [3.64.193.173] INFO[0116] [sidekick] Sidekick container already created on host [18.185.105.131] INFO[0116] Image [rancher/hyperkube:v1.18.16-rancher1] exists on host [3.64.193.173] INFO[0116] Image [rancher/hyperkube:v1.18.16-rancher1] exists on host [18.185.105.131] INFO[0116] [sidekick] Sidekick container already created on host [18.195.249.125] INFO[0116] Image [rancher/hyperkube:v1.18.16-rancher1] exists on host [18.195.249.125] INFO[0116] Starting container [kubelet] on host [3.64.193.173], try #1 INFO[0116] Starting container [kubelet] on host [18.185.105.131], try #1 INFO[0116] Starting container [kubelet] on host [18.195.249.125], try #1 INFO[0116] [worker] Successfully started [kubelet] container on host [3.64.193.173] INFO[0116] [healthcheck] Start Healthcheck on service [kubelet] on host [3.64.193.173] INFO[0116] [worker] Successfully started [kubelet] container on host [18.185.105.131] INFO[0116] [healthcheck] Start Healthcheck on service [kubelet] on host [18.185.105.131] INFO[0116] [worker] Successfully started [kubelet] container on host [18.195.249.125] INFO[0116] [healthcheck] Start Healthcheck on service [kubelet] on host [18.195.249.125] INFO[0121] [healthcheck] service [kubelet] on host [3.64.193.173] is healthy INFO[0121] Image [rancher/rke-tools:v0.1.72] exists on host [3.64.193.173] INFO[0121] [healthcheck] service [kubelet] on host [18.185.105.131] is healthy INFO[0121] Image [rancher/rke-tools:v0.1.72] exists on host [18.185.105.131] INFO[0121] [healthcheck] service [kubelet] on host [18.195.249.125] is healthy INFO[0121] Image [rancher/rke-tools:v0.1.72] exists on host [18.195.249.125] INFO[0121] Starting container [rke-log-linker] on host [3.64.193.173], try #1 INFO[0121] Starting container [rke-log-linker] on host [18.185.105.131], try #1 INFO[0121] Starting container [rke-log-linker] on host [18.195.249.125], try #1 INFO[0121] [worker] Successfully started [rke-log-linker] container on host [3.64.193.173] INFO[0121] Removing container [rke-log-linker] on host [3.64.193.173], try #1 INFO[0121] [worker] Successfully started [rke-log-linker] container on host [18.185.105.131] INFO[0121] Removing container [rke-log-linker] on host [18.185.105.131], try #1 INFO[0122] [worker] Successfully started [rke-log-linker] container on host [18.195.249.125] INFO[0122] Removing container [rke-log-linker] on host [18.195.249.125], try #1 INFO[0122] [remove/rke-log-linker] Successfully removed container on host [3.64.193.173] INFO[0122] Image [rancher/hyperkube:v1.18.16-rancher1] exists on host [3.64.193.173] INFO[0122] [remove/rke-log-linker] Successfully removed container on host [18.185.105.131] INFO[0122] Image [rancher/hyperkube:v1.18.16-rancher1] exists on host [18.185.105.131] INFO[0122] Starting container [kube-proxy] on host [3.64.193.173], try #1 INFO[0122] [remove/rke-log-linker] Successfully removed container on host [18.195.249.125] INFO[0122] Starting container [kube-proxy] on host [18.185.105.131], try #1 INFO[0122] Image [rancher/hyperkube:v1.18.16-rancher1] exists on host [18.195.249.125] INFO[0122] [worker] Successfully started [kube-proxy] container on host [3.64.193.173] INFO[0122] [healthcheck] Start Healthcheck on service [kube-proxy] on host [3.64.193.173] INFO[0122] Starting container [kube-proxy] on host [18.195.249.125], try #1 INFO[0122] [worker] Successfully started [kube-proxy] container on host [18.185.105.131] INFO[0122] [healthcheck] Start Healthcheck on service [kube-proxy] on host [18.185.105.131] INFO[0122] [worker] Successfully started [kube-proxy] container on host [18.195.249.125] INFO[0122] [healthcheck] Start Healthcheck on service [kube-proxy] on host [18.195.249.125] INFO[0127] [healthcheck] service [kube-proxy] on host [3.64.193.173] is healthy INFO[0127] Image [rancher/rke-tools:v0.1.72] exists on host [3.64.193.173] INFO[0127] [healthcheck] service [kube-proxy] on host [18.185.105.131] is healthy INFO[0127] Image [rancher/rke-tools:v0.1.72] exists on host [18.185.105.131] INFO[0127] Starting container [rke-log-linker] on host [3.64.193.173], try #1 INFO[0127] [healthcheck] service [kube-proxy] on host [18.195.249.125] is healthy INFO[0127] Image [rancher/rke-tools:v0.1.72] exists on host [18.195.249.125] INFO[0127] Starting container [rke-log-linker] on host [18.185.105.131], try #1 INFO[0127] Starting container [rke-log-linker] on host [18.195.249.125], try #1 INFO[0128] [worker] Successfully started [rke-log-linker] container on host [3.64.193.173] INFO[0128] Removing container [rke-log-linker] on host [3.64.193.173], try #1 INFO[0128] [worker] Successfully started [rke-log-linker] container on host [18.185.105.131] INFO[0128] Removing container [rke-log-linker] on host [18.185.105.131], try #1 INFO[0128] [worker] Successfully started [rke-log-linker] container on host [18.195.249.125] INFO[0128] Removing container [rke-log-linker] on host [18.195.249.125], try #1 INFO[0128] [remove/rke-log-linker] Successfully removed container on host [3.64.193.173] INFO[0128] [remove/rke-log-linker] Successfully removed container on host [18.185.105.131] INFO[0128] [remove/rke-log-linker] Successfully removed container on host [18.195.249.125] INFO[0128] [worker] Successfully started Worker Plane.. INFO[0128] Image [rancher/rke-tools:v0.1.72] exists on host [3.64.193.173] INFO[0128] Image [rancher/rke-tools:v0.1.72] exists on host [18.185.105.131] INFO[0128] Image [rancher/rke-tools:v0.1.72] exists on host [18.195.249.125] INFO[0128] Starting container [rke-log-cleaner] on host [18.185.105.131], try #1 INFO[0128] Starting container [rke-log-cleaner] on host [18.195.249.125], try #1 INFO[0128] Starting container [rke-log-cleaner] on host [3.64.193.173], try #1 INFO[0129] [cleanup] Successfully started [rke-log-cleaner] container on host [18.185.105.131] INFO[0129] Removing container [rke-log-cleaner] on host [18.185.105.131], try #1 INFO[0129] [cleanup] Successfully started [rke-log-cleaner] container on host [18.195.249.125] INFO[0129] Removing container [rke-log-cleaner] on host [18.195.249.125], try #1 INFO[0129] [cleanup] Successfully started [rke-log-cleaner] container on host [3.64.193.173] INFO[0129] Removing container [rke-log-cleaner] on host [3.64.193.173], try #1 INFO[0129] [remove/rke-log-cleaner] Successfully removed container on host [18.185.105.131] INFO[0129] [remove/rke-log-cleaner] Successfully removed container on host [18.195.249.125] INFO[0129] [remove/rke-log-cleaner] Successfully removed container on host [3.64.193.173] INFO[0129] [sync] Syncing nodes Labels and Taints INFO[0129] [sync] Successfully synced nodes Labels and Taints INFO[0129] [network] Setting up network plugin: canal INFO[0129] [addons] Saving ConfigMap for addon rke-network-plugin to Kubernetes INFO[0129] [addons] Successfully saved ConfigMap for addon rke-network-plugin to Kubernetes INFO[0129] [addons] Executing deploy job rke-network-plugin INFO[0134] [addons] Setting up coredns INFO[0134] [addons] Saving ConfigMap for addon rke-coredns-addon to Kubernetes INFO[0134] [addons] Successfully saved ConfigMap for addon rke-coredns-addon to Kubernetes INFO[0134] [addons] Executing deploy job rke-coredns-addon INFO[0139] [addons] CoreDNS deployed successfully INFO[0139] [dns] DNS provider coredns deployed successfully INFO[0139] [addons] Setting up Metrics Server INFO[0139] [addons] Saving ConfigMap for addon rke-metrics-addon to Kubernetes INFO[0139] [addons] Successfully saved ConfigMap for addon rke-metrics-addon to Kubernetes INFO[0139] [addons] Executing deploy job rke-metrics-addon INFO[0144] [addons] Metrics Server deployed successfully INFO[0144] [ingress] Setting up nginx ingress controller INFO[0144] [addons] Saving ConfigMap for addon rke-ingress-controller to Kubernetes INFO[0144] [addons] Successfully saved ConfigMap for addon rke-ingress-controller to Kubernetes INFO[0144] [addons] Executing deploy job rke-ingress-controller INFO[0149] [ingress] ingress controller nginx deployed successfully INFO[0149] [addons] Setting up user addons INFO[0149] [addons] no user addons defined INFO[0149] Finished building Kubernetes cluster successfully
If all went fine you should have a three node cluster after a few minutes:
rancher@rancher1:~$ export KUBECONFIG=kube_config_cluster.yml rancher@rancher1:~$ kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME rancher2 Ready controlplane,etcd,worker 4m51s v1.18.16 10.0.1.253 Debian GNU/Linux 10 (buster) 4.19.0-14-cloud-amd64 docker://19.3.15 rancher1 Ready controlplane,etcd,worker 4m51s v1.18.16 10.0.1.168 Debian GNU/Linux 10 (buster) 4.19.0-14-cloud-amd64 docker://19.3.15 rancher3 Ready controlplane,etcd,worker 4m51s v1.18.16 10.0.1.73 Debian GNU/Linux 10 (buster) 4.19.0-14-cloud-amd64 docker://19.3.15
Again, very easy to setup. We still do not have Rancher running, just RKE in a three node configuration. The installation of Rancher itself will be the topic for the next post.