Tuesday, December 11, 2018

How to setup rancher 2.x cluster ( with 3 nodes example )

0.prepare rancher controller nodes

Here we use 3 nodes as rancher High Availability ( HA ) installation.

For creating a VM I was using Proxmox but it can be any VM solution such as VirtualBox or VMWare ... etc.

Then you build your VM base on Rancher OS (ROS - https://github.com/rancher/os/releases). I try CentOS and Ubuntu OS, however, you need to deal with those firewall rules (https://rancher.com/docs/rancher/v2.x/en/installation/references/) ... etc which is kinds of lousy.


Then you prepare a `cloud-config.yml` file the example can be as below.

#cloud-config hostname: {{ hostname }} rancher:
network:
interfaces:
eth0:
address: {{ ip_address }}/{{ subnet_mask}}
gateway: {{ gateway }}
dns:
nameservers:
- {{ nameserver_1 }}
- {{ nameserver_2 }}
- {{ nameserver_3 }} ssh_authorized_keys:
- {{ ssh_key_public }}

$ sudo ros install -c cloud-config.yml -d /dev/sda

Then you can jump from your management note with the key you put in cloud-config.yml

$ ssh rancher@<rancher node ip>

# e.g. ssh rancher@192.168.60.44

1.Create Nodes and LB

Here, we try ngnix on e.g. 192.168.1.100 - Ref: https://rancher.com/docs/rancher/v2.x/en/installation/ha/create-nodes-lb/nginx/

a. install NGNIX 

Ref: https://www.nginx.com/resources/wiki/start/topics/tutorials/install/ but in my test case I use alpine Ref: https://wiki.alpinelinux.org/wiki/Nginx

b. create NGINX Configuration 

e.g. ngnix.conf in /etc/nginx/nginx.conf
From nginx.conf, replace <IP_NODE_1>, <IP_NODE_2>, and <IP_NODE_3> with the IPs of your node.

worker_processes 4;
worker_rlimit_nofile 40000;

events {
    worker_connections 8192;
}

http {
    server {
        listen         80;
        return 301 https://$host$request_uri;
    }
}

stream {
    upstream rancher_servers {
        least_conn;
        server <IP_NODE_1>:443 max_fails=3 fail_timeout=5s;
        server <IP_NODE_2>:443 max_fails=3 fail_timeout=5s;
        server <IP_NODE_3>:443 max_fails=3 fail_timeout=5s;
    }
    server {
        listen     443;
        proxy_pass rancher_servers;
    }
}

Save nginx.conf to your load balancer at the following path: /etc/nginx/nginx.conf.

c. restart ngnix

$ nginx -s reload

2.Install k8s with RKE

Ref: https://rancher.com/docs/rancher/v2.x/en/installation/ha/kubernetes-rke/

a. Create the rancher-cluster.yml file

The contenct template can look like as below.

nodes:
  - address: 165.227.114.63
    internal_address: 172.16.22.12
    user: ubuntu
    role: [controlplane,worker,etcd]
  - address: 165.227.116.167
    internal_address: 172.16.32.37
    user: ubuntu
    role: [controlplane,worker,etcd]
  - address: 165.227.127.226
    internal_address: 172.16.42.73
    user: ubuntu
    role: [controlplane,worker,etcd]

services:
  etcd:
    snapshot: true
    creation: 6h
    retention: 24h


here is our real example
nodes:
  - address: 192.168.1.101
    internal_address: 192.168.1.101
    user: rancher
    role: [controlplane,worker,etcd]
  - address: 192.168.1.102
    internal_address: 192.168.1.102
    user: rancher
    role: [controlplane,worker,etcd]
  - address: 192.168.1.103
    internal_address: 192.168.1.103
    user: rancher
    role: [controlplane,worker,etcd]

services:
  etcd:
    snapshot: true
    creation: 6h
    retention: 24h

b. Install RKE

Ref: https://rancher.com/docs/rke/v0.1.x/en/installation/
PS: we prepare RKE to install on sac-mgmt since it has all the ssh key in ros node. For install RKE it required ssh privilege, however after RKE install k8s, we can leverage k8s API server to run kubectl and helm from the client ( your mac or ubuntu ).

download rke from https://github.com/rancher/rke/releases

$ wget https://github.com/rancher/rke/releases/download/v0.1.13/rke_darwin-amd64

mv name

# MacOS
$ mv rke_darwin-amd64 rke
# Linux
$ mv rke_linux-amd64 rke
# Windows PowerShell
> mv rke_windows-amd64.exe rke.exe

make executable

$ chmod +x rke

double check via checkin rke version

$ ./rke --version

c. Run RKE

$ rke up --config ./rancher-cluster.yml

After installation doen you should be able to see kube_config_rancher-cluster.yml. The content looks like this , you can copy it into your local

$ cat kube_config_rancher-cluster.yml
apiVersion: v1
kind: Config
clusters:
- cluster:
    api-version: v1
    certificate-authority-data: xxx
    server: "https://192.168.1.101:6443"
  name: "local"
contexts:
- context:
    cluster: "local"
    user: "kube-admin-local"
  name: "local"
current-context: "local"
users:
- name: "kube-admin-local"
  user:
    client-certificate-data: xxx
    client-key-data: xxx

d. Test Cluster

Test cluster can be done by your client( mac or Ubuntu PC). First of all, you need to install kubectl.
Ref: https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl

For Mac: https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-with-homebrew-on-macos
$ brew install kubernetes-cli

For Ubuntu:
sudo apt-get update && sudo apt-get install -y apt-transport-https
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubectl

Then you need to set configuration
You can copy this file to $HOME/.kube/config or if you are working with multiple Kubernetes clusters, set the KUBECONFIG environmental variable to the path of kube_config_rancher-cluster.yml.

$ export KUBECONFIG=$(pwd)/kube_config_rancher-cluster.yml

$ kubectl get nodes
NAME             STATUS    AGE       VERSION
192.168.1.101   Ready     2d        v1.11.5
192.168.1.102   Ready     2d        v1.11.5
192.168.1.103   Ready     2d        v1.11.5

Try kubectl get all

$ kubectl get all --all-namespaces
NAMESPACE       NAME                                       READY     STATUS    RESTARTS   AGE
cattle-system   po/cattle-cluster-agent-5cfc664c96-zgz64   1/1       Running   0          2d
cattle-system   po/cattle-node-agent-dgb6x                 1/1       Running   0          2d
cattle-system   po/cattle-node-agent-p8c8s                 1/1       Running   0          2d
cattle-system   po/cattle-node-agent-vwlfl                 1/1       Running   0          2d
cattle-system   po/rancher-f4fb5f5c6-79qf9                 1/1       Running   7          2d
cattle-system   po/rancher-f4fb5f5c6-7nrz5                 1/1       Running   4          2d
cattle-system   po/rancher-f4fb5f5c6-xn4kc                 1/1       Running   6          2d
ingress-nginx   po/default-http-backend-797c5bc547-8jtnq   1/1       Running   0          2d
ingress-nginx   po/nginx-ingress-controller-kg9dp          1/1       Running   0          2d
ingress-nginx   po/nginx-ingress-controller-r9kcj          1/1       Running   0          2d
ingress-nginx   po/nginx-ingress-controller-znjs8          1/1       Running   0          2d
kube-system     po/canal-hjw45                             3/3       Running   0          2d
kube-system     po/canal-pm5hq                             3/3       Running   0          2d
kube-system     po/canal-qxgkk                             3/3       Running   0          2d
kube-system     po/cert-manager-7d4bfc44ff-x6v2k           1/1       Running   0          2d
kube-system     po/kube-dns-7588d5b5f5-gd4br               3/3       Running   0          2d
kube-system     po/kube-dns-autoscaler-5db9bbb766-2g528    1/1       Running   0          2d
kube-system     po/metrics-server-97bc649d5-pqdg8          1/1       Running   0          2d
kube-system     po/tiller-deploy-7b6f5d9dbc-76qxf          1/1       Running   0          2d

NAMESPACE       NAME                       CLUSTER-IP     EXTERNAL-IP   PORT(S)         AGE
cattle-system   svc/rancher                10.43.253.92   <none>        80/TCP          2d
default         svc/kubernetes             10.43.0.1      <none>        443/TCP         2d
ingress-nginx   svc/default-http-backend   10.43.46.215   <none>        80/TCP          2d
kube-system     svc/kube-dns               10.43.0.10     <none>        53/UDP,53/TCP   2d
kube-system     svc/metrics-server         10.43.3.216    <none>        443/TCP         2d
kube-system     svc/tiller-deploy          10.43.72.38    <none>        44134/TCP       2d

NAMESPACE     NAME                                     DESIRED   SUCCESSFUL   AGE
kube-system   jobs/rke-ingress-controller-deploy-job   1         1            2d
kube-system   jobs/rke-kubedns-addon-deploy-job        1         1            2d
kube-system   jobs/rke-metrics-addon-deploy-job        1         1            2d
kube-system   jobs/rke-network-plugin-deploy-job       1         1            2d

NAMESPACE       NAME                          DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
cattle-system   deploy/cattle-cluster-agent   1         1         1            1           2d
cattle-system   deploy/rancher                3         3         3            3           2d
ingress-nginx   deploy/default-http-backend   1         1         1            1           2d
kube-system     deploy/cert-manager           1         1         1            1           2d
kube-system     deploy/kube-dns               1         1         1            1           2d
kube-system     deploy/kube-dns-autoscaler    1         1         1            1           2d
kube-system     deploy/metrics-server         1         1         1            1           2d
kube-system     deploy/tiller-deploy          1         1         1            1           2d

NAMESPACE       NAME                                 DESIRED   CURRENT   READY     AGE
cattle-system   rs/cattle-cluster-agent-5cfc664c96   1         1         1         2d
cattle-system   rs/rancher-f4fb5f5c6                 3         3         3         2d
ingress-nginx   rs/default-http-backend-797c5bc547   1         1         1         2d
kube-system     rs/cert-manager-7d4bfc44ff           1         1         1         2d
kube-system     rs/kube-dns-7588d5b5f5               1         1         1         2d
kube-system     rs/kube-dns-autoscaler-5db9bbb766    1         1         1         2d
kube-system     rs/metrics-server-97bc649d5          1         1         1         2d
kube-system     rs/tiller-deploy-7b6f5d9dbc          1         1         1         2d

3.Initialize Helm (Install Tiller)

Ref: https://rancher.com/docs/rancher/v2.x/en/installation/ha/helm-init/

a. install tiller on k8s cluster

$ kubectl -n kube-system create serviceaccount tiller

$ kubectl create clusterrolebinding tiller \
  --clusterrole cluster-admin \
  --serviceaccount=kube-system:tiller

$ helm init --service-account tiller

b. test your tiller via k8s

kubectl -n kube-system  rollout status deploy/tiller-deploy
Waiting for deployment "tiller-deploy" rollout to finish: 0 of 1 updated replicas are available...
deployment "tiller-deploy" successfully rolled out
c. test your tiller via helm
Before you test with helm, you need to install helm Ref: https://rancher.com/docs/rancher/v2.x/en/installation/ha/helm-init/
Or you just download binary should work still Prefer way - Ref: https://github.com/helm/helm/releases

PS: watch out the helm vs tiller version match, in our prod we need to use helm v2.8 only since newest v2.12 miss matches the tiller which is installed by rancher.

$ ./darwin-amd64/helm version
Client: &version.Version{SemVer:"v2.8.2", GitCommit:"a80231648a1473929271764b920a8e346f6de844", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.8.0", GitCommit:"14af25f1de6832228539259b821949d20069a222", GitTreeState:"clean"}

4.Install Rancher

Ref: https://rancher.com/docs/rancher/v2.x/en/installation/ha/helm-rancher/

a. Add helm chart repo for rancher

Use helm repo add the command to add the Helm chart repository that contains charts to install Rancher. For more information about the repository choices and which is best for your use case, see Choosing a Version of Rancher.

Replace both occurences of <CHART_REPO> with the Helm chart repository that you want to use (i.e. latest or stable).
helm repo add rancher-<CHART_REPO> https://releases.rancher.com/server-charts/<CHART_REPO>

$ ./helm repo add rancher-stable 
https://releases.rancher.com/server-charts/stable

$ ./helm repo list
NAME          URL
stable        https://kubernetes-charts.storage.googleapis.com
local          http://127.0.0.1:8879/charts
rancher-stable https://releases.rancher.com/server-charts/stable

b. Choose SSL configuration.

This example uses cert-manager

install cert-manager

$ helm install stable/cert-manager \
  --name cert-manager \
  --namespace kube-system

wait cert-manger roll out

$ kubectl -n kube-system rollout status deploy/cert-manager
Waiting for deployment "cert-manager" rollout to finish: 0 of 1 updated replicas are available...
deployment "cert-manager" successfully rolled out

double check cert-manager

$ ./helm list
NAME        REVISION UPDATED                  STATUS  CHART              NAMESPACE
cert-manager 1        Sat Dec  8 04:32:41 2018 DEPLOYED cert-manager-v0.5.2 kube-system

c. Install Rancher


install rancher

$ ./helm install rancher-stable/rancher   --name rancher   --namespace cattle-system   --set hostname=rancher2.swiftstack.org

wait for rancher roll out

$ kubectl -n cattle-system rollout status deploy/rancher
Waiting for deployment "rancher" rollout to finish: 0 of 3 updated replicas are available...
deployment "rancher" successfully rolled out

double check rancher

$ ./helm list
NAME        REVISION UPDATED                  STATUS  CHART              NAMESPACE
cert-manager 1        Sat Dec  8 04:32:41 2018 DEPLOYED cert-manager-v0.5.2 kube-system
rancher      1        Sat Dec  8 04:35:50 2018 DEPLOYED rancher-2018.12.1  cattle-system

Try using our own Cert files

set up secret - Ref: https://rancher.com/docs/rancher/v2.x/en/installation/ha/helm-rancher/tls-secrets/

PS: Combine the server certificate followed by an intermediate certificate(s) needed into a file named tls.crt. Copy your certificate key into a file named tls.key.

$ kubectl -n cattle-system create secret tls tls-rancher-ingress \
  --cert=tls.crt \
  --key=tls.key

Using a Private CA-Signed Certificate
If you are using a private CA, Rancher requires a copy of the CA certificate which is used by the Rancher Agent to validate the connection to the server.

Copy the CA certificate into a file named cacerts.pem and use kubectl to create the tls-ca secret in the cattle-system namespace.
Important: Make sure the file is called cacerts.pem as Rancher uses that filename to configure the CA certificate.

$ kubectl -n cattle-system create secret generic tls-ca \
  --from-file=cacerts.pem

Install with cert file

$ helm install rancher-<CHART_REPO>/rancher \
  --name rancher \
  --namespace cattle-system \
  --set hostname=rancher.my.org \
  --set ingress.tls.source=secret

wait for rancher roll out

kubectl -n cattle-system rollout status deploy/rancher
Waiting for deployment "rancher" rollout to finish: 0 of 3 updated replicas are available...
deployment "rancher" successfully rolled out

double check

kubectl -n cattle-system get deploy rancher
NAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
rancher   3         3         3            3           3m

5.Remove Rancher

a. remove rancher helm

$ ./helm del --purge rancher

b. remove k8s cluster

$ rke remove --config rancher-cluster.yml

c. reinstall ros 

just repeat section 0.prepare rancher controller nodes

Reference
https://rancher.com/docs/rancher/v2.x/en/installation/ha/

1 comment: