Johnny ( Chianing ) Wang : December 2018

0.prepare rancher controller nodes

Here we use 3 nodes as rancher High Availability ( HA ) installation.

For creating a VM I was using Proxmox but it can be any VM solution such as VirtualBox or VMWare ... etc.

Then you build your VM base on Rancher OS (ROS - https://github.com/rancher/os/releases). I try CentOS and Ubuntu OS, however, you need to deal with those firewall rules (https://rancher.com/docs/rancher/v2.x/en/installation/references/) ... etc which is kinds of lousy.

Then you prepare a `cloud-config.yml` file the example can be as below.

#cloud-config
hostname: {{ hostname }}
rancher:
  network:
    interfaces:
      eth0:
        address: {{ ip_address }}/{{ subnet_mask}}
        gateway: {{ gateway }}
    dns:
      nameservers:
        - {{ nameserver_1 }} 
        - {{ nameserver_2 }}
        - {{ nameserver_3 }}
ssh_authorized_keys:
  - {{ ssh_key_public }}

$ sudo ros install -c cloud-config.yml -d /dev/sda

Then you can jump from your management note with the key you put in cloud-config.yml

$ ssh rancher@<rancher node ip>

# e.g. ssh rancher@192.168.60.44

1.Create Nodes and LB

Here, we try ngnix on e.g. 192.168.1.100 - Ref: https://rancher.com/docs/rancher/v2.x/en/installation/ha/create-nodes-lb/nginx/

a. install NGNIX

Ref: https://www.nginx.com/resources/wiki/start/topics/tutorials/install/ but in my test case I use alpine Ref: https://wiki.alpinelinux.org/wiki/Nginx

b. create NGINX Configuration

e.g. ngnix.conf in /etc/nginx/nginx.conf
From nginx.conf, replace <IP_NODE_1>, <IP_NODE_2>, and <IP_NODE_3> with the IPs of your node.

worker_processes 4;
worker_rlimit_nofile 40000;

events {
worker_connections 8192;
}

http {
server {
listen 80;
return 301 https://$host$request_uri;
}
}

stream {
upstream rancher_servers {
least_conn;
server <IP_NODE_1>:443 max_fails=3 fail_timeout=5s;
server <IP_NODE_2>:443 max_fails=3 fail_timeout=5s;
server <IP_NODE_3>:443 max_fails=3 fail_timeout=5s;
}
server {
listen 443;
proxy_pass rancher_servers;
}
}

Save nginx.conf to your load balancer at the following path: /etc/nginx/nginx.conf.

c. restart ngnix

$ nginx -s reload

2.Install k8s with RKE

Ref: https://rancher.com/docs/rancher/v2.x/en/installation/ha/kubernetes-rke/

a. Create the rancher-cluster.yml file

The contenct template can look like as below.

nodes:
- address: 165.227.114.63
internal_address: 172.16.22.12
user: ubuntu
role: [controlplane,worker,etcd]
- address: 165.227.116.167
internal_address: 172.16.32.37
user: ubuntu
role: [controlplane,worker,etcd]
- address: 165.227.127.226
internal_address: 172.16.42.73
user: ubuntu
role: [controlplane,worker,etcd]

services:
etcd:
snapshot: true
creation: 6h
retention: 24h

here is our real example
nodes:
- address: 192.168.1.101
internal_address: 192.168.1.101
user: rancher
role: [controlplane,worker,etcd]
- address: 192.168.1.102
internal_address: 192.168.1.102
user: rancher
role: [controlplane,worker,etcd]
- address: 192.168.1.103
internal_address: 192.168.1.103
user: rancher
role: [controlplane,worker,etcd]

services:
etcd:
snapshot: true
creation: 6h
retention: 24h

b. Install RKE

Ref: https://rancher.com/docs/rke/v0.1.x/en/installation/
PS: we prepare RKE to install on sac-mgmt since it has all the ssh key in ros node. For install RKE it required ssh privilege, however after RKE install k8s, we can leverage k8s API server to run kubectl and helm from the client ( your mac or ubuntu ).

download rke from https://github.com/rancher/rke/releases

$ wget https://github.com/rancher/rke/releases/download/v0.1.13/rke_darwin-amd64

mv name

# MacOS
$ mv rke_darwin-amd64 rke
# Linux
$ mv rke_linux-amd64 rke
# Windows PowerShell
> mv rke_windows-amd64.exe rke.exe

make executable

$ chmod +x rke

double check via checkin rke version

$ ./rke --version

c. Run RKE

$ rke up --config ./rancher-cluster.yml

After installation doen you should be able to see kube_config_rancher-cluster.yml. The content looks like this , you can copy it into your local

$ cat kube_config_rancher-cluster.yml
apiVersion: v1
kind: Config
clusters:
- cluster:
api-version: v1
certificate-authority-data: xxx
server: "https://192.168.1.101:6443"
name: "local"
contexts:
- context:
cluster: "local"
user: "kube-admin-local"
name: "local"
current-context: "local"
users:
- name: "kube-admin-local"
user:
client-certificate-data: xxx
client-key-data: xxx

d. Test Cluster

Test cluster can be done by your client( mac or Ubuntu PC). First of all, you need to install kubectl.
Ref: https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl

For Mac: https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-with-homebrew-on-macos
$ brew install kubernetes-cli

For Ubuntu:
sudo apt-get update && sudo apt-get install -y apt-transport-https
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubectl

Then you need to set configuration
You can copy this file to $HOME/.kube/config or if you are working with multiple Kubernetes clusters, set the KUBECONFIG environmental variable to the path of kube_config_rancher-cluster.yml.

$ export KUBECONFIG=$(pwd)/kube_config_rancher-cluster.yml

$ kubectl get nodes
NAME STATUS AGE VERSION
192.168.1.101 Ready 2d v1.11.5
192.168.1.102 Ready 2d v1.11.5
192.168.1.103 Ready 2d v1.11.5

Try kubectl get all

$ kubectl get all --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
cattle-system po/cattle-cluster-agent-5cfc664c96-zgz64 1/1 Running 0 2d
cattle-system po/cattle-node-agent-dgb6x 1/1 Running 0 2d
cattle-system po/cattle-node-agent-p8c8s 1/1 Running 0 2d
cattle-system po/cattle-node-agent-vwlfl 1/1 Running 0 2d
cattle-system po/rancher-f4fb5f5c6-79qf9 1/1 Running 7 2d
cattle-system po/rancher-f4fb5f5c6-7nrz5 1/1 Running 4 2d
cattle-system po/rancher-f4fb5f5c6-xn4kc 1/1 Running 6 2d
ingress-nginx po/default-http-backend-797c5bc547-8jtnq 1/1 Running 0 2d
ingress-nginx po/nginx-ingress-controller-kg9dp 1/1 Running 0 2d
ingress-nginx po/nginx-ingress-controller-r9kcj 1/1 Running 0 2d
ingress-nginx po/nginx-ingress-controller-znjs8 1/1 Running 0 2d
kube-system po/canal-hjw45 3/3 Running 0 2d
kube-system po/canal-pm5hq 3/3 Running 0 2d
kube-system po/canal-qxgkk 3/3 Running 0 2d
kube-system po/cert-manager-7d4bfc44ff-x6v2k 1/1 Running 0 2d
kube-system po/kube-dns-7588d5b5f5-gd4br 3/3 Running 0 2d
kube-system po/kube-dns-autoscaler-5db9bbb766-2g528 1/1 Running 0 2d
kube-system po/metrics-server-97bc649d5-pqdg8 1/1 Running 0 2d
kube-system po/tiller-deploy-7b6f5d9dbc-76qxf 1/1 Running 0 2d

NAMESPACE NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
cattle-system svc/rancher 10.43.253.92 <none> 80/TCP 2d
default svc/kubernetes 10.43.0.1 <none> 443/TCP 2d
ingress-nginx svc/default-http-backend 10.43.46.215 <none> 80/TCP 2d
kube-system svc/kube-dns 10.43.0.10 <none> 53/UDP,53/TCP 2d
kube-system svc/metrics-server 10.43.3.216 <none> 443/TCP 2d
kube-system svc/tiller-deploy 10.43.72.38 <none> 44134/TCP 2d

NAMESPACE NAME DESIRED SUCCESSFUL AGE
kube-system jobs/rke-ingress-controller-deploy-job 1 1 2d
kube-system jobs/rke-kubedns-addon-deploy-job 1 1 2d
kube-system jobs/rke-metrics-addon-deploy-job 1 1 2d
kube-system jobs/rke-network-plugin-deploy-job 1 1 2d

NAMESPACE NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
cattle-system deploy/cattle-cluster-agent 1 1 1 1 2d
cattle-system deploy/rancher 3 3 3 3 2d
ingress-nginx deploy/default-http-backend 1 1 1 1 2d
kube-system deploy/cert-manager 1 1 1 1 2d
kube-system deploy/kube-dns 1 1 1 1 2d
kube-system deploy/kube-dns-autoscaler 1 1 1 1 2d
kube-system deploy/metrics-server 1 1 1 1 2d
kube-system deploy/tiller-deploy 1 1 1 1 2d

NAMESPACE NAME DESIRED CURRENT READY AGE
cattle-system rs/cattle-cluster-agent-5cfc664c96 1 1 1 2d
cattle-system rs/rancher-f4fb5f5c6 3 3 3 2d
ingress-nginx rs/default-http-backend-797c5bc547 1 1 1 2d
kube-system rs/cert-manager-7d4bfc44ff 1 1 1 2d
kube-system rs/kube-dns-7588d5b5f5 1 1 1 2d
kube-system rs/kube-dns-autoscaler-5db9bbb766 1 1 1 2d
kube-system rs/metrics-server-97bc649d5 1 1 1 2d
kube-system rs/tiller-deploy-7b6f5d9dbc 1 1 1 2d

3.Initialize Helm (Install Tiller)

Ref: https://rancher.com/docs/rancher/v2.x/en/installation/ha/helm-init/

a. install tiller on k8s cluster

$ kubectl -n kube-system create serviceaccount tiller

$ kubectl create clusterrolebinding tiller \
--clusterrole cluster-admin \
--serviceaccount=kube-system:tiller

$ helm init --service-account tiller

b. test your tiller via k8s

kubectl -n kube-system rollout status deploy/tiller-deploy
Waiting for deployment "tiller-deploy" rollout to finish: 0 of 1 updated replicas are available...
deployment "tiller-deploy" successfully rolled out
c. test your tiller via helm
Before you test with helm, you need to install helm Ref: https://rancher.com/docs/rancher/v2.x/en/installation/ha/helm-init/
Or you just download binary should work still Prefer way - Ref: https://github.com/helm/helm/releases

PS: watch out the helm vs tiller version match, in our prod we need to use helm v2.8 only since newest v2.12 miss matches the tiller which is installed by rancher.

$ ./darwin-amd64/helm version
Client: &version.Version{SemVer:"v2.8.2", GitCommit:"a80231648a1473929271764b920a8e346f6de844", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.8.0", GitCommit:"14af25f1de6832228539259b821949d20069a222", GitTreeState:"clean"}

4.Install Rancher

Ref: https://rancher.com/docs/rancher/v2.x/en/installation/ha/helm-rancher/

a. Add helm chart repo for rancher

Use helm repo add the command to add the Helm chart repository that contains charts to install Rancher. For more information about the repository choices and which is best for your use case, see Choosing a Version of Rancher.

Replace both occurences of <CHART_REPO> with the Helm chart repository that you want to use (i.e. latest or stable).
helm repo add rancher-<CHART_REPO> https://releases.rancher.com/server-charts/<CHART_REPO>

$ ./helm repo add rancher-stable
https://releases.rancher.com/server-charts/stable

$ ./helm repo list
NAME URL
stable https://kubernetes-charts.storage.googleapis.com
local http://127.0.0.1:8879/charts
rancher-stable https://releases.rancher.com/server-charts/stable

b. Choose SSL configuration.

This example uses cert-manager

install cert-manager

$ helm install stable/cert-manager \
--name cert-manager \
--namespace kube-system

wait cert-manger roll out

$ kubectl -n kube-system rollout status deploy/cert-manager
Waiting for deployment "cert-manager" rollout to finish: 0 of 1 updated replicas are available...
deployment "cert-manager" successfully rolled out

double check cert-manager

$ ./helm list
NAME REVISION UPDATED STATUS CHART NAMESPACE
cert-manager 1 Sat Dec 8 04:32:41 2018 DEPLOYED cert-manager-v0.5.2 kube-system

c. Install Rancher

install rancher

$ ./helm install rancher-stable/rancher --name rancher --namespace cattle-system --set hostname=rancher2.swiftstack.org

wait for rancher roll out

$ kubectl -n cattle-system rollout status deploy/rancher
Waiting for deployment "rancher" rollout to finish: 0 of 3 updated replicas are available...
deployment "rancher" successfully rolled out

double check rancher

$ ./helm list
NAME REVISION UPDATED STATUS CHART NAMESPACE
cert-manager 1 Sat Dec 8 04:32:41 2018 DEPLOYED cert-manager-v0.5.2 kube-system
rancher 1 Sat Dec 8 04:35:50 2018 DEPLOYED rancher-2018.12.1 cattle-system

Try using our own Cert files

set up secret - Ref: https://rancher.com/docs/rancher/v2.x/en/installation/ha/helm-rancher/tls-secrets/

PS: Combine the server certificate followed by an intermediate certificate(s) needed into a file named tls.crt. Copy your certificate key into a file named tls.key.

$ kubectl -n cattle-system create secret tls tls-rancher-ingress \
--cert=tls.crt \
--key=tls.key

Using a Private CA-Signed Certificate
If you are using a private CA, Rancher requires a copy of the CA certificate which is used by the Rancher Agent to validate the connection to the server.

Copy the CA certificate into a file named cacerts.pem and use kubectl to create the tls-ca secret in the cattle-system namespace.
Important: Make sure the file is called cacerts.pem as Rancher uses that filename to configure the CA certificate.

$ kubectl -n cattle-system create secret generic tls-ca \
--from-file=cacerts.pem

Install with cert file

$ helm install rancher-<CHART_REPO>/rancher \
--name rancher \
--namespace cattle-system \
--set hostname=rancher.my.org \
--set ingress.tls.source=secret

wait for rancher roll out

kubectl -n cattle-system rollout status deploy/rancher
Waiting for deployment "rancher" rollout to finish: 0 of 3 updated replicas are available...
deployment "rancher" successfully rolled out

double check

kubectl -n cattle-system get deploy rancher
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
rancher 3 3 3 3 3m

5.Remove Rancher

a. remove rancher helm

$ ./helm del --purge rancher

b. remove k8s cluster

$ rke remove --config rancher-cluster.yml

c. reinstall ros

just repeat section 0.prepare rancher controller nodes

Reference
https://rancher.com/docs/rancher/v2.x/en/installation/ha/

Johnny ( Chianing ) Wang

Tuesday, December 11, 2018

How to setup rancher 2.x cluster ( with 3 nodes example )