Tuesday, January 16, 2018

How to containerized OpenStack Swift into multi-nodes

Docker Openstack Swift ( almost all Versions ) Multi-Nodes

This repository contains Dockerfile for creating openstack swift cluster(pike as example) in multi-nodes ( proxy and storage node). Storage node includes account, container and object node. There is two Dockerfiles one for proxy and another storage for creating images. The files folder contains related configuration files of proxy and storage node.
  1. docker build proxy and storage image (Use Ubuntu 14.04 and Ubuntu 16.04 for installing docker.)
  2. docker run to create container base on proxy and storage image
  3. docker exec to start supervisord to start required services.

The swift version we support

For Ubuntu 14.04 (trusty) base docker image:

We can support swift version as below only for Ubuntu 14.04.
  • juno
  • kilo
  • liberty
  • mitaka
# Make sure you are using ubuntu 1404 base docker image
e.g in Dockerfile
FROM ubuntu:14.04

# you need to replace Dockerfile from juno to the version you like
e.g in Dockerfile
RUN sudo echo "deb http://ubuntu-cloud.archive.canonical.com/ubuntu" "trusty-updates/juno main" > /etc/apt/sources.list.d/cloudarchive-juno.list
to
RUN sudo echo "deb http://ubuntu-cloud.archive.canonical.com/ubuntu" "trusty-updates/mitaka main" > /etc/apt/sources.list.d/cloudarchive-mitaka.list

For Ubuntu 16.04 (xenial) base docker image

We can support swift version as below only for Ubuntu 16.04.
  • newton
  • ocata
  • pike
  • queens
# Make sure you are using ubuntu 1404 base docker image
e.g in Dockerfile
FROM ubuntu:16.04

# you need to replace Dockerfile from queens to the version you like
e.g in Dockerfile
RUN sudo echo "deb http://ubuntu-cloud.archive.canonical.com/ubuntu" "xenial-updates/newton main" > /etc/apt/sources.list.d/cloudarchive-newton.list
to
RUN sudo echo "deb http://ubuntu-cloud.archive.canonical.com/ubuntu" "xenial-updates/ocata main" > /etc/apt/sources.list.d/cloudarchive-ocata.list

# BUT for pike and queens, We have difficulty for just use RUN sudo echo "deb ... ", we hve to use add-apt-repository. The detail you can check in Dockerfile.
e.g in Dockerfile
add-apt-repository cloud-archive:pike -y
apt update -y && apt dist-upgrade -y

For Proxy Docker Container

Docker Build proxy image with verion ( tag= or dev, test ... etc )

Inside files folder there is ring files (*.ring.gz and *.builder), we should only need *.ring.gz but still provide *.builder in case you need any modification (https://docs.openstack.org/mitaka/install-guide-rdo/swift-initial-rings.html or https://docs.openstack.org/swift/latest/ring_partpower.html)
e.g. IP address in configuration files related to swift proxy should be replaced. Or replace ring files as your swift cluster is configured and make changes in all configuration files as per your requirement. Change bind_ip inside proxy-server.conf file. After replacing and configure the proxy related file go to the directory which contains Docker file and run following command for creating proxy image.
cd ./proxy
docker build -t="swift-proxy:pike" .
  • Use docker images to double check

Docker Run create proxy docker container

It will take some time for downloading and configuring image. After completing process for creating proxy docker container for swift object storage. Run below command for creating proxy container. Host networking mode is needed for this container to work, so the --net=host parameter must be used when starting up an instance of this container.
docker run -i -t -d --net=host --hostname="proxy_test" --name="proxy_test" swift-proxy:pike
  • Use docker ps -a to double check

Docker execute to start supervisord for proxy docker container

Run the below command for start the all services of proxy node. Here we are using supervisor for controlling all services because docker container runs a single process when it is launched.
docker exec -it proxy_test /bin/sh -c /usr/local/bin/start.sh
Before your run above docker exec, you will see netstat as below.
# netstat -ntlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1230/sshd
tcp6       0      0 :::22                   :::*                    LISTEN      1230/sshd
After your run above docker exec, you should be able to see proxy(8080) and memcache(11211) port shows up
# netstat -ntlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:11211           0.0.0.0:*               LISTEN      17217/memcached
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN      17216/python
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1230/sshd
tcp6       0      0 :::11211                :::*                    LISTEN      17217/memcached
tcp6       0      0 :::22                   :::*                    LISTEN      1230/sshd

For Storage Docker Container

Docker Build storage image with verion ( tag )

Replace all ring files inside files directory and configure all configuration files ( if you have different device layout ) and run the following command for creating image.
cd ../storage/
docker build -t="swift-storage:pike" .
  • Use docker images to double check

Docker Run create storage docker container

In order for Docker to use whole file systems from its host OS, those file systems need to be mounted on the host OS, and they need to be passed to the container in the form of -v parameters. So create drives and write the file system over that drive and mount it. For running storage container run following command
docker run -i -t -d --net=host --hostname="storage_test" --name="storage_test" -v /srv/node/sdb1:/srv/node/sdb1:rw swift-storage:pike
  • Use docker ps -a to double check

Docker execute to start supervisord for storage docker container

Run the below command for start the all services of proxy node. Here we are using supervisor for controlling all services because docker container runs a single process when it is launched.
docker exec -it storage_test /bin/sh -c /usr/local/bin/start.sh
Before your run above docker exec, you will see netstat as below.
# netstat -ntlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:11211           0.0.0.0:*               LISTEN      17217/memcached
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN      17216/python
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1230/sshd
tcp6       0      0 :::11211                :::*                    LISTEN      17217/memcached
tcp6       0      0 :::22                   :::*                    LISTEN      1230/sshd
After your run above docker exec, you should be able to see account(6002), container(6001) and object(6000) port shows up
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:11211           0.0.0.0:*               LISTEN      17217/memcached
tcp        0      0 127.0.0.1:6000          0.0.0.0:*               LISTEN      17784/python
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN      17216/python
tcp        0      0 127.0.0.1:6001          0.0.0.0:*               LISTEN      17780/python
tcp        0      0 127.0.0.1:6002          0.0.0.0:*               LISTEN      17787/python
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1230/sshd
tcp6       0      0 :::11211                :::*                    LISTEN      17217/memcached
tcp6       0      0 :::22                   :::*                    LISTEN      1230/sshd

For AccountContainer Docker Container

Docker Build accountcontainer image with verion ( tag= or dev, test ... etc)

Inside files folder there is ring files (*.ring.gz and *.builder), we should only need *.ring.gz but still provide *.builder in case you need any modification (https://docs.openstack.org/mitaka/install-guide-rdo/swift-initial-rings.html or https://docs.openstack.org/swift/latest/ring_partpower.html)
e.g. IP address in configuration files related to swift accountcontainer should be replaced. Or replace ring files as your swift cluster is configured and make changes in all configuration files as per your requirement. Change bind_ip inside account-server.conf and container-server.conf file. After replacing and configure the account and container docker node, related files go to the directory which contains Docker file and run following command for creating accountcontainer image.
cd ./accountcontainer
docker build -t="swift-accountcontainer:queens" .
  • Use docker images to double check

Docker Run create accountcontainer docker container

It will take some time for downloading and configuring image, if you don't have base image in your local yet. After completing process for creating accountcontainer docker container for swift object storage. Run below command for creating accountcontainer docker container. Host networking mode is needed for this container to work, so the --net=host parameter must be used when starting up an instance of this container.
docker run -i -t -d --net=host --hostname="accountcontainer_test" --name="accountcontainer_test" swift-accountcontainer:pike
  • Use docker ps -a to double check

Docker execute to start supervisord for accountcontainer docker container

Run the below command for start the all services of accountcontainer node. Here we are using supervisor for controlling all services because docker container runs a single process when it is launched.
docker exec -it accountcontainer_test /bin/sh -c /usr/local/bin/start.sh
Before your run above docker exec, you will see netstat as below.
# netstat -ntlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1230/sshd
tcp6       0      0 :::22                   :::*                    LISTEN      1230/sshd
After your run above docker exec, you should be able to see account(6002) and container(6001) port shows up
# netstat -ntlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:6001          0.0.0.0:*               LISTEN      17780/python
tcp        0      0 127.0.0.1:6002          0.0.0.0:*               LISTEN      17787/python
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1230/sshd
tcp6       0      0 :::22                   :::*                    LISTEN      1230/sshd

For Object Docker Container

Docker Build object image with verion ( tag= or dev, test ... etc)

Inside files folder there is ring files (*.ring.gz and *.builder), we should only need *.ring.gz but still provide *.builder in case you need any modification (https://docs.openstack.org/mitaka/install-guide-rdo/swift-initial-rings.html or https://docs.openstack.org/swift/latest/ring_partpower.html)
e.g. IP address in configuration files related to swift object should be replaced. Or replace ring files as your swift cluster is configured and make changes in all configuration files as per your requirement. Change bind_ip inside object-server.conf file. After replacing and configure the account and container docker node, related files go to the directory which contains Docker file and run following command for creating object image.
cd ./object
docker build -t="swift-object:queens" .
  • Use docker images to double check

Docker Run create object docker container

It will take some time for downloading and configuring image, if you don't have base image in your local yet. After completing process for creating object docker container for swift object storage. Run below command for creating object docker container. Host networking mode is needed for this container to work, so the --net=host parameter must be used when starting up an instance of this container.
docker run -i -t -d --net=host --hostname="object_test" --name="object_test" swift-object:pike
  • Use docker ps -a to double check

Docker execute to start supervisord for object docker container

Run the below command for start the all services of object node. Here we are using supervisor for controlling all services because docker container runs a single process when it is launched.
docker exec -it object_test /bin/sh -c /usr/local/bin/start.sh
Before your run above docker exec, you will see netstat as below.
# netstat -ntlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1230/sshd
tcp6       0      0 :::22                   :::*                    LISTEN      1230/sshd
After your run above docker exec, you should be able to see object(6000) port shows up
# netstat -ntlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:6000          0.0.0.0:*               LISTEN      17780/python
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1230/sshd
tcp6       0      0 :::22                   :::*                    LISTEN      1230/sshd

For PACO (SAIO Swift All in One) Docker Container

Docker Build swift paco image with verion ( tag )

Replace all ring files inside files directory and configure all configuration files ( if you have different device layout ) and run the following command for creating image.
cd ./paco/
docker build -t="swift-paco:queens" .
  • Use docker images to double check

Docker Run create swift paco docker container

In order for Docker to use whole file systems from its host OS, those file systems need to be mounted on the host OS, and they need to be passed to the container in the form of -v parameters. So create drives and write the file system over that drive and mount it. For running storage container run following command
docker run -i -t -d --net=host --hostname="paco_test" --name="paco_test" -v /srv/node/sdb1:/srv/node/sdb1:rw swift-paco:queens
  • Use docker ps -a to double check
  • PS: since we are using --net=host, thus if you are running proxy_test or storage_test container which are using those port on host, docker run might fail since the ports are in use. You need either docker rm those containers or use port mapping when docker run.

Docker execute to start supervisord for swift paco docker container

Run the below command for start the all services of proxy node. Here we are using supervisor for controlling all services because docker container runs a single process when it is launched.
docker exec -it paco_test /bin/sh -c /usr/local/bin/start.sh
Before your run above docker exec, you will see netstat (-ntlp or -tunapls) as below.
# netstat -ntlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1230/sshd
tcp6       0      0 :::22                   :::*                    LISTEN      1230/sshd
After your run above docker exec, you should be able to see account(6002), container(6001) and object(6000) port shows up
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:11211           0.0.0.0:*               LISTEN      17217/memcached
tcp        0      0 127.0.0.1:6000          0.0.0.0:*               LISTEN      17784/python
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN      17216/python
tcp        0      0 127.0.0.1:6001          0.0.0.0:*               LISTEN      17780/python
tcp        0      0 127.0.0.1:6002          0.0.0.0:*               LISTEN      17787/python
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1230/sshd
tcp6       0      0 :::11211                :::*                    LISTEN      17217/memcached
tcp6       0      0 :::22                   :::*                    LISTEN      1230/sshd

Integration Test

Integration Test manually via python-swiftclient

Test is pretty starightforward just use the cli as below.
# swift -A http://127.0.0.1:8080/auth/v1.0 -U test:tester -K testing stat -v
                     StorageURL: http://127.0.0.1:8080/v1/AUTH_test
                     Auth Token: AUTH_tk24351ad1cc9248a3852abfb7faa80e71
                        Account: AUTH_test
                     Containers: 1
                        Objects: 0
                          Bytes: 0
Containers in policy "policy-0": 1
   Objects in policy "policy-0": 0
     Bytes in policy "policy-0": 0
                    X-Timestamp: 1516138923.98611
                     X-Trans-Id: txf98fb97425e94fd48e360-005a5e802f
                   Content-Type: text/plain; charset=utf-8
                  Accept-Ranges: bytes

Preparation - setup.sh

sudo apt-get update -y && apt-get install -y apt-transport-https
sudo apt install docker.io -y
sudo systemctl start docker
sudo systemctl enable docker
sudo apt install python-swiftclient -y

PS

Ensure proper ownership to all folders which related to storage container. You can use the command as below to jump into container always
# for proxy node container
docker exec -it proxy_test /bin/bash
# for storage node container
docker exec -it storage_test /bin/bash
doube check as below
# for storage node
chown -R swift:swift /srv/node
# for proxy and storage node
chown -R swift:swift /etc/swift
Or Run the below command for start all services related to proxy or storage container.
# for proxy and storage node
service supervisor restart
# for storage node ( it will fail if the port are in use )
swift-init all restart

Reference:
https://github.com/chianingwang/docker-swift-in-multinodes
on No comments:
Labels: , , , , , , , ,
Subscribe to: Posts (Atom)