Monday, September 21, 2015

How to generate tempurl for Swift object in Openstack.

tempurl is a good feature when you are using object storage, since you can generate a link and share your object to the people quick. They can download your object (file) directly from the link without credentail within the expiration time.

However, I was struggling for tempurl for pass days. Because during the days, I was dispirited and can't find too much logical info. People said about tempurl is more complicated than what's I thought and didn't explain the basic logic but just the cli (command lines).

Other than that, the some reference links includes the unit test cli but didn't give an overall summary. Here, I will explain the how curl and swift-client works to operate the tempurl and give a unit test example.

In sum, I really want give a quick intro about the basic logic, and after you aware of it, the cli for you will be logical as nature. Here are the items I would like to talk.
  1. tempurl basic logic with python example
  2. tempurl Lab Setup 
    1. Server: Swift AIO setup
    2. Client: curl / swift-client setup
  3. tempurl configuration.
  4. tempurl unit test
  5. swift over swift vs swift over ceph.
    1. swift on ceph won't work regarding the limitation

1. tempurl basic logic with python example

For generate the tempurl from OpenStack Swift, the basic idea is you have a ID ( key ) with your account then you know where is your object your would like to share with other people. After that you hash the key and a expiration date/time with the link ( url ). That's it !
  1. Key
  2. Path
  3. Expiration Date/Time

python example

The python example as below explain all the logics.












The info you need to assign.
  1. method: eg: GET
  2. host: if you test in your POC SAIO VM, you can give "http://127.0.0.1:8080"
  3. expiration: give the seconds for temp url expired.
  4. path: the path for your object, eg: '/v1/AUTH_test/testCon/test.txt'
  5. key: the key you would like to have or assign, eg: 'secret'
  6. sig: generate signature for combining 1, 2, 3 and 4
  7. tempurl: combine with hostname, object directory and temp_url_sign
--------python code start--------
'''
Created on Sep 21, 2015

@author: johnnywang
'''

import hmac
from hashlib import sha1
from time import time

#1
method = 'GET'

#2
host = "http://127.0.0.1:8080"

#3
duration_in_seconds = 6000  # Duration for which the url is valid
expires = int(time() + duration_in_seconds)

#4
path = '/v1/AUTH_test/testCon/test.txt'

#5
key = 'secret'

hmac_body = '%s\n%s\n%s' % (method, expires, path)
hmac_body = hmac.new(key, hmac_body, sha1).hexdigest()

#6
sig = hmac.new(key, hmac_body, sha1).hexdigest()

#7
rest_uri = "{host}{path}?temp_url_sig={sig}&temp_url_expires={expires}".format(
            host=host, path=path, sig=sig, expires=expires)
print rest_uri

--------python code end--------

output:

http://127.0.0.1:8080/v1/AUTH_test/testCon/test.txt?temp_url_sig=efbd2cad15e098d2327b8c7109886882f4a7afec&temp_url_expires=1442870614

2. tempurl Lab setup ( Swift over Swift )

SAIO setup - Server

SAIO is swift all in one lab, you can follow the links as below I present before, setup a POC lab for verify tempurl quick. It's quick and easy.
PS: you can configure port forwarding then you can ssh to your virtualbox SAIO vm.


There has two major client tools which can operate the swift, there are curl and swift-client.


curl setup - Client

in ubuntu

#sudo apt-get install curl libcurl3 libcurl3-dev

in mac

#brew install curl


swift-client - Client

in ubuntu

#sudo aptitude install python-pip
#sudo pip install python-swiftclient

in mac

#sudo easy_install pip
#sudo pip install --upgrade setuptools
#sudo pip install python-swiftclient

 http://thornelabs.net/2014/10/29/installing-python-swiftclient-on-os-x-yosemite.html 

3. tempurl configuration

The key for make tempurl works is inject a key or second key in account level metadata. You can use curl or swift-client. Both should be all working well.

Get token and endpoint url via Swift-client


  • Get Auth-token or you username and password directly
swift@swift-VirtualBox:~$ swift -A http://127.0.0.1:8080/auth/v1.0 -U swift -K swift auth
export OS_STORAGE_URL=http://127.0.0.1:8080/v1/AUTH_swift
export OS_AUTH_TOKEN=AUTH_tk5050d2a92c43422e871d60cffa309022

swift@swift-VirtualBox:~$ swift -A http://127.0.0.1:8080/auth/v1.0 -U swift -K swift auth -v
export ST_AUTH=http://127.0.0.1:8080/auth/v1.0
export ST_USER=swift
export ST_KEY=swift

  • Once you get the endpoint and token you can insert a key into account's metadata (X-Account-Meta-Temp-Url-Key: secret)
  • Or you can access swift via username and password directly.
swift@swift-VirtualBox:~$ swift -A http://127.0.0.1:8080/auth/v1.0 -U test:tester -K testing stat
                    Account: AUTH_test
                 Containers: 2
                    Objects: 0
                      Bytes: 0
Containers in policy "gold": 2
   Objects in policy "gold": 0
     Bytes in policy "gold": 0
          Meta Temp-Url-Key: secret
                X-Timestamp: 1439949170.11303
                 X-Trans-Id: tx81e1acbb06f544af90429-0055d5654d
               Content-Type: text/plain; charset=utf-8

              Accept-Ranges: bytes

Inject tempurl key



using swift-client

 post "Temp-Url-Key"
swift@swift-VirtualBox:~$ swift --os-auth-token AUTH_tk29ec321ad87b43d0bfd8a7b687ab4a2f --os-storage-url http://127.0.0.1:8080/v1/AUTH_test post -m "Temp-URL-Key: secret"

 Double check

 using curl with token
swift@swift-VirtualBox:~$ curl -v -H 'X-Auth-Token: AUTH_tk71106acb07784da1859cd2e434eba109' http://127.0.0.1:8080/v1/AUTH_test/
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> GET /v1/AUTH_test/ HTTP/1.1
> User-Agent: curl/7.35.0
> Host: 127.0.0.1:8080
> Accept: */*
> X-Auth-Token: AUTH_tk71106acb07784da1859cd2e434eba109
> 
< HTTP/1.1 200 OK
< X-Account-Storage-Policy-Gold-Bytes-Used: 0
< Content-Length: 19
< X-Account-Storage-Policy-Gold-Object-Count: 0
< X-Account-Object-Count: 0
< X-Timestamp: 1439949170.11303
< X-Account-Meta-Temp-Url-Key: secret
< X-Account-Storage-Policy-Gold-Container-Count: 2
< X-Account-Bytes-Used: 0
< X-Account-Container-Count: 2
< Content-Type: text/plain; charset=utf-8
< Accept-Ranges: bytes
< X-Trans-Id: tx1d08eb3202cb44a49bc6e-0055d409fd
< Date: Wed, 19 Aug 2015 04:45:49 GMT
< 
testCon
testFolder
* Connection #0 to host 127.0.0.1 left intact

swift@swift-VirtualBox:~$ 

 via token
swift@swift-VirtualBox:~$ swift --os-auth-token AUTH_tk29ec321ad87b43d0bfd8a7b687ab4a2f --os-storage-url http://127.0.0.1:8080/v1/AUTH_test stat
                    Account: AUTH_test
                 Containers: 2
                    Objects: 0
                      Bytes: 0
Containers in policy "gold": 2
   Objects in policy "gold": 0
     Bytes in policy "gold": 0
          Meta Temp-Url-Key: secret
                X-Timestamp: 1439949170.11303
                 X-Trans-Id: tx2a025cd404df4b8c9c8c4-0055d55f12
               Content-Type: text/plain; charset=utf-8
              Accept-Ranges: bytes

 via username/password
swift@swift-VirtualBox:~$ swift -A http://127.0.0.1:8080/auth/v1.0 -U test:tester -K testing stat
                    Account: AUTH_test
                 Containers: 2
                    Objects: 0
                      Bytes: 0
Containers in policy "gold": 2
   Objects in policy "gold": 0
     Bytes in policy "gold": 0
          Meta Temp-Url-Key: secret
                X-Timestamp: 1439949170.11303
                 X-Trans-Id: tx81e1acbb06f544af90429-0055d5654d
               Content-Type: text/plain; charset=utf-8

              Accept-Ranges: bytes

As long as you can see the key in account level metadata, you can use the python script in above to put the path for the object and generate the tempurl.

 PS: above is V1 authentication, here is V2 example if your keystone server is running V2 authentication.
swift@swift-VirtualBox:~$ swift -V 2.0 -A http://127.0.0.1:8080/auth/v2.0 -U swift -K swift post -m Temp-URL-Key:secret

swift@swift-VirtualBox:~$ swift -V 2.0 -A http://127.0.0.1:8080/auth/v2.0 -U swift -K swift stat -v

4. tempurl unit test

Here is the example you can use tempcurl to download the file, or you can copy the whole tempurl hyper-link and put on any browser to get the file.

swift@swift-VirtualBox:~$ curl -L -o 'download.txt' 'http://127.0.0.1:8080/v1/AUTH_test/testCon/test.txt?temp_url_sig=0003e838e9a253780e14b9a38545333c3255c119&temp_url_expires=1440014448'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    15  100    15    0     0    450      0 --:--:-- --:--:-- --:--:--   483

swift@swift-VirtualBox:~$ cat download.txt 
This is a TEST

PS: since download link with hash, you might want to use curl -L to trust the hash as a part of url, and -O is output into what's the name you would like to have for the object content.

5. tempurl for Swift over Ceph 

Except the swift over swift, you can run swift over ceph which means leverage radosgateway for accessing your object in ceph. For POC, you can setup a Ceph lab follow these links.
tempurl available in RGW (http://ceph.com/docs/master/radosgw/swift/tempurl). unfortunately, it's not working in my lap or the environment I aware. 

This is what’s I realized. For temp url, we need to inject a temp-url-key in account metadata, but in Swift over ceph (http://ceph.com/docs/master/radosgw/swift/) API, it doesn’t seem to allow us to change it but only read it.  Without temp-url-key in account metadata, the tempurl will never work. 

Feature
Status
Remarks
Authentication
Supported

Get Account Metadata
Supported
No custom metadata
Swift ACLs
Supported
Supports a subset of Swift ACLs
List Containers
Supported

Delete Container
Supported

Create Container
Supported

Get Container Metadata
Supported

Update Container Metadata
Supported

Delete Container Metadata
Supported

List Objects
Supported

Static Website
Not Supported

Create Object
Supported

Create Large Object
Supported

Delete Object
Supported

Get Object
Supported

Copy Object
Supported

Get Object Metadata
Supported

Update Object Metadata
Supported

Expiring Objects
Not Supported

Object Versioning
Not Supported

CORS
Not Supported


So if you know above statement is wrong or you figure the way out, please share with me.

Reference:

on
Labels: , , , , , , , , ,

1 comment:

  1. nuwan.agSeptember 29, 2017 at 10:42 PM

    Thanks! This article really helped with figuring out how to work with the temp-url key.

    ReplyDelete

Subscribe to: Post Comments (Atom)