Tuesday, November 28, 2017

How to setup Varnish to connect Object Storage with SSL

This article will like to show you how to leverage caching to speed up your Transaction. The solution we pick is Varnish however, varnish doesn't support ssl unless you pay for Varnish +. Thus we take advantage reverse proxy + stunnel to make client -> ssl -> varnish --> ssl > swift ( object storage ) happen.

Idea

  Client -> SSL(443) ->
               Ngnix   |
               (80)  <-
              Varnish
              (8080) ->
              Stunnel  |   
                        -> SSL (443) Swift Endpoint

Outline

  • Setup / Configure Nginx Reverse Proxy with SSL ( 443 to 80 )
  • Setup / Configure Varnish for cache ( 80 to 8080 )
  • Setup / Configure Stunnel for SSL tunnel ( 8080 to 443 )
PS: if you would like to let Varnish connect to Swift Endpoint with SSL directly, you would need to buy Varnish-plus Or the option could be connecting with Swift Endpoint without SSL.


Setup / Configure Nginx Reverse Proxy with SSL ( 443 to 80 )

Here I use wildcard cert "*.ss.org", ss02 is swift endpoint, ss03 is nginx, varnish and stunnel server.
$ apt install nginx -y

comment out /etc/nginx/sites-available/default 
since we don't need default web page run on port 80


$ vi /etc/ngnix/sites-enabled/default

server {
        listen 443 ssl http2 default_server;
       listen [::]:443 ssl http2 default_server;

        server_name <your ngnix + varnish + stunnel hostname>
        ssl_prefer_server_ciphers  on;
        ssl_ciphers  'ECDH !aNULL !eNULL !SSLv2 !SSLv3';
        ssl_certificate /etc/nginx/ssl/star_ss_org.pem;
        ssl_certificate_key /etc/nginx/ssl/star_ss_org.key;

        location / {
            proxy_pass http://127.0.0.1;
            proxy_set_header X-Real-IP  $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto https;
            proxy_set_header X-Forwarded-Port 443;
            proxy_set_header Host $host;
        }
}
$ move your *pem, *key under /etc/nginx/ssl/ PS; if you don't have then mkdir /etc/nginx/ssl
service nginx restart

Setup / Configure Varnish for cache ( 80 to 8080 )

$ apt install varnish -y


/etc/default/varnish. In the file you'll find some text that looks like this:
DAEMON_OPTS="-a :6081 \
             -T localhost:6082 \
             -f /etc/varnish/default.vcl \
             -S /etc/varnish/secret \
             -s default,256m"
Change it to:
DAEMON_OPTS="-a :80 \
             -T localhost:6082 \
             -f /etc/varnish/default.vcl \
             -S /etc/varnish/secret \
             -s default,256m"

/etc/varnish/default.vcl
vcl 4.0;
# List of upstream proxies we trust to set X-Forwarded-For correctly.
acl upstream_proxy {
     "127.0.0.1";
}

# Default backend definition. Set this to point to your content server.
backend default {
    .host = "127.0.0.1";
    .port = "8080";
}

sub vcl_recv {
    # Set the X-Forwarded-For header so the backend can see the original
    # IP address. If one is already set by an upstream proxy, we'll just re-use that.
    if (client.ip ~ upstream_proxy && req.http.X-Forwarded-For) {
        set req.http.X-Forwarded-For = req.http.X-Forwarded-For;
    } else {
        set req.http.X-Forwarded-For = regsub(client.ip, ":.*", "");
    }
}

sub vcl_hash {
    # URL and hostname/IP are the default components of the vcl_hash
    # implementation. We add more below.
    hash_data(req.url);
    if (req.http.host) {
        hash_data(req.http.host);
    } else {
        hash_data(server.ip);
    }

    # Include the X-Forward-Proto header, since we want to treat HTTPS
    # requests differently, and make sure this header is always passed
    # properly to the backend server.
    if (req.http.X-Forwarded-Proto) {
        hash_data(req.http.X-Forwarded-Proto);
    }
    #return (hash);
}

sub vcl_backend_response {
    # Happens after we have read the response headers from the backend.
    #
    # Here you clean the response headers, removing silly Set-Cookie headers
    # and other mistakes your backend does.
    set beresp.ttl = 60s;
}
restart varnish service
$ service varnish restart
you can double check via
$ varnishlog or varnishstat

Setup / Configure Stunnel for SSL tunnel ( 8080 to 443 )

$ apt-get install stunnel4 -y

$ cat /etc/nginx/ssl/star_ss_org.key /etc/nginx/ssl/star_ss_org.pem >> /etc/nginx/ssl/stunnel.pem

$ vi /etc/stunnel/stunnel.conf
client = yes
[varnish]
accept = 8080
connect = <swift endpoint>:443 # e.g ss02.ss.org
cert = /etc/nginx/ssl/stunnel.pem
restart stunnel service
$ service stunnel4 restart

Final test

Test for your clients
$ swift --debug -A https://ss03.ss.org/auth/v1.0 -U ss -K ss stat -v
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): ss03.ss.org
DEBUG:requests.packages.urllib3.connectionpool:"GET /auth/v1.0 HTTP/1.1" 200 0
DEBUG:swiftclient:REQ: curl -i https://ss03.ss.org/auth/v1.0 -X GET
DEBUG:swiftclient:RESP STATUS: 200 OK
DEBUG:swiftclient:RESP HEADERS: {u'Content-Length': u'0', u'X-Varnish': u'32912', u'Set-Cookie': u'X-Auth-Token=AUTH_tk7b9e4ac141dc4cd3b202189214ca7d05; Path=/', u'Age': u'0', u'X-Trans-Id': u'tx6fb7e69b98a64cf1b1c02-005a0b88db', u'Server': u'nginx/1.10.3 (Ubuntu)', u'Connection': u'keep-alive', u'Via': u'1.1 varnish-v4', u'Accept-Ranges': u'bytes', u'X-Storage-Token': u'AUTH_tk7b9e4ac141dc4cd3b202189214ca7d05', u'Date': u'Wed, 15 Nov 2017 01:05:32 GMT', u'X-Storage-Url': u'https://ss03.ss.org/v1/AUTH_ss', u'X-Auth-Token': u'AUTH_tk7b9e4ac141dc4cd3b202189214ca7d05', u'Content-Type': u'text/plain; charset=UTF-8', u'X-Openstack-Request-Id': u'tx6fb7e69b98a64cf1b1c02-005a0b88db'}
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): ss03.ss.org
DEBUG:requests.packages.urllib3.connectionpool:"HEAD /v1/AUTH_ss HTTP/1.1" 200 0
DEBUG:swiftclient:REQ: curl -i https://ss03.ss.org/v1/AUTH_ss -I -H "X-Auth-Token: AUTH_tk7b9e4ac141dc4cd3b202189214ca7d05"
DEBUG:swiftclient:RESP STATUS: 200 OK
DEBUG:swiftclient:RESP HEADERS: {u'X-Openstack-Request-Id': u'tx2b3cce143b4c459c864fe-005a0b88db', u'Content-Length': u'122', u'X-Account-Storage-Policy-Standard-Replica-Container-Count': u'4', u'Age': u'0', u'Accept-Ranges': u'bytes', u'X-Account-Storage-Policy-Standard-Replica-Object-Count': u'11', u'Via': u'1.1 varnish-v4', u'X-Account-Bytes-Used': u'1155343', u'Server': u'nginx/1.10.3 (Ubuntu)', u'Connection': u'keep-alive', u'X-Varnish': u'108', u'X-Timestamp': u'1510255401.40488', u'X-Account-Meta-Temp-Url-Key': u'380a104a-6b76-4b4f-8588-1c02ff3b25cc', u'X-Trans-Id': u'tx2b3cce143b4c459c864fe-005a0b88db', u'Date': u'Wed, 15 Nov 2017 01:05:32 GMT', u'X-Account-Storage-Policy-Standard-Replica-Bytes-Used': u'1155343', u'X-Account-Container-Count': u'4', u'Content-Type': u'text/plain; charset=utf-8', u'X-Account-Object-Count': u'11'}
                             StorageURL: https://ss03.ss.org/v1/AUTH_ss
                             Auth Token: AUTH_tk7b9e4ac141dc4cd3b202189214ca7d05
                                Account: AUTH_ss
                             Containers: 4
                                Objects: 11
                                  Bytes: 1155343
Containers in policy "standard-replica": 4
   Objects in policy "standard-replica": 11
     Bytes in policy "standard-replica": 1155343
                      Meta Temp-Url-Key: 380a104a-6b76-4b4f-8588-1c02ff3b25cc
                 X-Openstack-Request-Id: tx2b3cce143b4c459c864fe-005a0b88db
                                    Via: 1.1 varnish-v4
                          Accept-Ranges: bytes
                                 Server: nginx/1.10.3 (Ubuntu)
                                    Age: 0
                             Connection: keep-alive
                              X-Varnish: 108
                            X-Timestamp: 1510255401.40488
                             X-Trans-Id: tx2b3cce143b4c459c864fe-005a0b88db
                           Content-Type: text/plain; charset=utf-8

You might see some varnish info attached to the header.
    Server: nginx/1.10.3 (Ubuntu)
       Age: 0
Connection: keep-alive
 X-Varnish: 108


From ngnix, varnish and stunnel node.
ngnix access or error log
$ tail -f /var/log/nginx/<node name>.access.log
or
$ tail -f /var/log/nginx/<node name>.access.log
varnish log
$ varnishlog

*   << BeReq    >> 109
-   Begin          bereq 108 fetch
-   Timestamp      Start: 1510707932.316241 0.000000 0.000000
-   BereqMethod    HEAD
-   BereqURL       /v1/AUTH_ss
-   BereqProtocol  HTTP/1.0
-   BereqHeader    X-Real-IP: 172.28.128.1
-   BereqHeader    X-Forwarded-Proto: https
-   BereqHeader    X-Forwarded-Port: 443
-   BereqHeader    Host: ss03.ss.org
-   BereqHeader    user-agent: python-swiftclient-3.4.0
-   BereqHeader    x-auth-token: AUTH_tk7b9e4ac141dc4cd3b202189214ca7d05
-   BereqHeader    X-Forwarded-For: 172.28.128.1, 127.0.0.1
-   BereqMethod    GET
-   BereqProtocol  HTTP/1.1
-   BereqHeader    Accept-Encoding: gzip
-   BereqHeader    X-Varnish: 109
-   VCL_call       BACKEND_FETCH
-   VCL_return     fetch
-   BackendOpen    18 boot.default 127.0.0.1 8080 127.0.0.1 60674
-   Timestamp      Bereq: 1510707932.316325 0.000084 0.000084
-   Timestamp      Beresp: 1510707932.336352 0.020111 0.020027
-   BerespProtocol HTTP/1.1
-   BerespStatus   200
-   BerespReason   OK
-   BerespHeader   Content-Length: 122
-   BerespHeader   X-Account-Container-Count: 4
-   BerespHeader   X-Account-Storage-Policy-Standard-Replica-Object-Count: 11
-   BerespHeader   X-Account-Object-Count: 11
-   BerespHeader   Accept-Ranges: bytes
-   BerespHeader   X-Timestamp: 1510255401.40488
-   BerespHeader   X-Account-Meta-Temp-Url-Key: 380a104a-6b76-4b4f-8588-1c02ff3b25cc
-   BerespHeader   X-Account-Storage-Policy-Standard-Replica-Bytes-Used: 1155343
-   BerespHeader   X-Account-Storage-Policy-Standard-Replica-Container-Count: 4
-   BerespHeader   Content-Type: text/plain; charset=utf-8
-   BerespHeader   X-Account-Bytes-Used: 1155343
-   BerespHeader   X-Trans-Id: tx2b3cce143b4c459c864fe-005a0b88db
-   BerespHeader   X-Openstack-Request-Id: tx2b3cce143b4c459c864fe-005a0b88db
-   BerespHeader   Date: Wed, 15 Nov 2017 00:22:51 GMT
-   TTL            RFC 120 10 -1 1510707932 1510707932 1510705371 0 0
-   VCL_call       BACKEND_RESPONSE
-   TTL            VCL 60 10 0 1510707932
-   VCL_return     deliver
-   Storage        malloc s0
-   ObjProtocol    HTTP/1.1
-   ObjStatus      200
-   ObjReason      OK
-   ObjHeader      Content-Length: 122
-   ObjHeader      X-Account-Container-Count: 4
-   ObjHeader      X-Account-Storage-Policy-Standard-Replica-Object-Count: 11
-   ObjHeader      X-Account-Object-Count: 11
-   ObjHeader      X-Timestamp: 1510255401.40488
-   ObjHeader      X-Account-Meta-Temp-Url-Key: 380a104a-6b76-4b4f-8588-1c02ff3b25cc
-   ObjHeader      X-Account-Storage-Policy-Standard-Replica-Bytes-Used: 1155343
-   ObjHeader      X-Account-Storage-Policy-Standard-Replica-Container-Count: 4
-   ObjHeader      Content-Type: text/plain; charset=utf-8
-   ObjHeader      X-Account-Bytes-Used: 1155343
-   ObjHeader      X-Trans-Id: tx2b3cce143b4c459c864fe-005a0b88db
-   ObjHeader      X-Openstack-Request-Id: tx2b3cce143b4c459c864fe-005a0b88db
-   ObjHeader      Date: Wed, 15 Nov 2017 00:22:51 GMT
-   Fetch_Body     3 length stream
-   BackendReuse   18 boot.default
-   Timestamp      BerespBody: 1510707932.336467 0.020226 0.000115
-   Length         122
-   BereqAcct      303 0 303 620 122 742
-   End

*   << Request  >> 108
-   Begin          req 107 rxreq
-   Timestamp      Start: 1510707932.316185 0.000000 0.000000
-   Timestamp      Req: 1510707932.316185 0.000000 0.000000
-   ReqStart       127.0.0.1 40464
-   ReqMethod      HEAD
-   ReqURL         /v1/AUTH_ss
-   ReqProtocol    HTTP/1.0
-   ReqHeader      X-Real-IP: 172.28.128.1
-   ReqHeader      X-Forwarded-For: 172.28.128.1
-   ReqHeader      X-Forwarded-Proto: https
-   ReqHeader      X-Forwarded-Port: 443
-   ReqHeader      Host: ss03.ss.org
-   ReqHeader      Connection: close
-   ReqHeader      Accept-Encoding: identity
-   ReqHeader      user-agent: python-swiftclient-3.4.0
-   ReqHeader      x-auth-token: AUTH_tk7b9e4ac141dc4cd3b202189214ca7d05
-   ReqUnset       X-Forwarded-For: 172.28.128.1
-   ReqHeader      X-Forwarded-For: 172.28.128.1, 127.0.0.1
-   VCL_call       RECV
-   VCL_acl        MATCH upstream_proxy "127.0.0.1"
-   ReqUnset       X-Forwarded-For: 172.28.128.1, 127.0.0.1
-   ReqHeader      X-Forwarded-For: 172.28.128.1, 127.0.0.1
-   VCL_return     hash
-   ReqUnset       Accept-Encoding: identity
-   VCL_call       HASH
-   VCL_return     lookup
-   VCL_call       MISS
-   VCL_return     fetch
-   Link           bereq 109 fetch
-   Timestamp      Fetch: 1510707932.336489 0.020304 0.020304
-   RespProtocol   HTTP/1.1
-   RespStatus     200
-   RespReason     OK
-   RespHeader     Content-Length: 122
-   RespHeader     X-Account-Container-Count: 4
-   RespHeader     X-Account-Storage-Policy-Standard-Replica-Object-Count: 11
-   RespHeader     X-Account-Object-Count: 11
-   RespHeader     X-Timestamp: 1510255401.40488
-   RespHeader     X-Account-Meta-Temp-Url-Key: 380a104a-6b76-4b4f-8588-1c02ff3b25cc
-   RespHeader     X-Account-Storage-Policy-Standard-Replica-Bytes-Used: 1155343
-   RespHeader     X-Account-Storage-Policy-Standard-Replica-Container-Count: 4
-   RespHeader     Content-Type: text/plain; charset=utf-8
-   RespHeader     X-Account-Bytes-Used: 1155343
-   RespHeader     X-Trans-Id: tx2b3cce143b4c459c864fe-005a0b88db
-   RespHeader     X-Openstack-Request-Id: tx2b3cce143b4c459c864fe-005a0b88db
-   RespHeader     Date: Wed, 15 Nov 2017 00:22:51 GMT
-   RespHeader     X-Varnish: 108
-   RespHeader     Age: 0
-   RespHeader     Via: 1.1 varnish-v4
-   VCL_call       DELIVER
-   VCL_return     deliver
-   Timestamp      Process: 1510707932.336530 0.020345 0.000041
-   RespHeader     Accept-Ranges: bytes
-   Debug          "RES_MODE 0"
-   RespHeader     Connection: close
-   Timestamp      Resp: 1510707932.336560 0.020375 0.000030
-   ReqAcct        300 0 300 684 0 684
-   End

*   << Session  >> 107
-   Begin          sess 0 HTTP/1
-   SessOpen       127.0.0.1 40464 :80 127.0.0.1 80 1510707932.316122 16
-   Link           req 108 rxreq
-   SessClose      RESP_CLOSE 0.020
-   End
-

可以學習的叫知識,可以練習的叫技巧,但學不來也練不會的,叫做熱情。
You can learn facts and you can train skills, but passion is something that has to be felt by the heart.
搖滾教室 (The School of Rock)

No comments:

Post a Comment