Wednesday, September 30, 2015

How to reference C++ lib in Java via JNI in OS X / Linux Ubuntu and Command Line / Eclipse

In my previous post, I was talking about the reference C++ dll in Windows (http://chianingwang.blogspot.com/2014/09/how-to-reference-c-dll-vs-2013-from.html). This post, I will share how to reference C++ lib in OS X. 

Again, the logic is pretty straightforward but the exception you might run into might take time to resolve it. Here is what's I try and I highlight them in yellow.


outline

  • Creating the Java Class for JNI shared ( function / arguments )
  • Compiling the Java Class for Generating C++ header 
  • Composing the C++ cpp or cc class
  • Sorting out the "jni.h" location as g++ reference
  • Compiling the C++ code for dynamic c lib
  • Run Java to call C++ Class
  • Using Eclipse in Ubuntu
    • C++ Shared Library Project
    • Java Project to run *.so


Creating the Java Class for JNI shared ( function / arguments )


We start by creating a Hello World java class with the C++ lib Class name we would like to have and method(function) we would like to use in C++ code.



$ vi HelloWorld.java

$ cat HelloWorld.java

class HelloWorld {

    private native void print();

    public static void main(String[] args) {

        new HelloWorld().print();

    }

    static {

        System.loadLibrary("HelloWorld");

    } 

}

This static section get executed first, which expects to load a JNI shared or dynamic library known as "HelloWorld".

PS: HelloWorld is C++ lib name, HelloWorld is C++ class name and print is the method(function) name.

Compiling the Java Class for Generating C++ header 

We need to compile the java class involves opening up the terminal and issuing the command as below, which is using javac to generate the class file that is needed to generate the appropriate JNI C++ classes.

$ javac HelloWorld.java

Creating the JNI C++ header, we can run the cli as below. I cat the C++ header content as below as well.


$ javah -jni HelloWorld

$ cat HelloWorld.h

/* DO NOT EDIT THIS FILE - it is machine generated */

#include <jni.h>

/* Header for class HelloWorld */



#ifndef _Included_HelloWorld

#define _Included_HelloWorld

#ifdef __cplusplus

extern "C" {

#endif

/*

 * Class:     HelloWorld

 * Method:    print

 * Signature: ()V

 */
JNIEXPORT void JNICALL Java_HelloWorld_print
  (JNIEnv *, jobject);

#ifdef __cplusplus
}
#endif
#endif

Composing the C++ cpp or cc class

C++ header is more like a window or declaration for your class and method(function) which let java know how to connect with C++ lib. After we got the C++ *.h, we need to compile C++ native code into dynamic library. (In OS X, jni shared/dymaic libraries have the extension .jnilib).


$ vi HelloWorld.cpp

$ cat HelloWorld.cpp

#include <jni.h>

#include <iostream>

#include "HelloWorld.h"

using namespace std;



JNIEXPORT void JNICALL

Java_HelloWorld_print(JNIEnv *, jobject){

    cout << "Oh Johnny, how handsome you are!\n";

    return;

}

Sorting out the "jni.h" location as g++ reference

In OS X, using finder to type jni.h and find the location for right click the file you found and display the info, then you will see the location for jni.h.




Compiling the C++ code for dynamic c lib

Generating Object File

To include the native code in the compilation of the java program, we need to compile native C++ code into dynamic library (OS X, jni shared/dynamic libraries have the extension .jnilib.) This is big different from the extensions on Windows and Solaris machine:, dll and so respectively.

We will need to leverage the jni.h to compile the C++ code into C++ *.o which is object file. Here, we include the -c option to generate the object file: HelloWorld.o from C++ code: HelloWorld.cpp, plus including jni.h path.



$ ls -lart

-rw-r--r--   1 johnnywa  staff    200 Sep 29 22:52 HelloWorld.java

-rw-r--r--   1 johnnywa  staff    377 Sep 29 22:53 HelloWorld.h

-rw-r--r--   1 johnnywa  staff    214 Sep 29 22:55 HelloWorld.cpp

-rw-r--r--   1 johnnywa  staff    442 Sep 30 11:57 HelloWorld.class

drwxr-xr-x  12 johnnywa  staff    408 Sep 30 12:47 .

$ g++ "-I/System/Library/Frameworks/JavaVM.framework/Versions/A/Headers" -c HelloWorld.cpp

$ ls -lart

-rw-r--r--   1 johnnywa  staff    200 Sep 29 22:52 HelloWorld.java

-rw-r--r--   1 johnnywa  staff    377 Sep 29 22:53 HelloWorld.h

-rw-r--r--   1 johnnywa  staff    214 Sep 29 22:55 HelloWorld.cpp

-rw-r--r--   1 johnnywa  staff    442 Sep 30 11:57 HelloWorld.class

-rw-r--r--   1 johnnywa  staff   6504 Sep 30 12:47 HelloWorld.o

drwxr-xr-x  13 johnnywa  staff    442 Sep 30 12:47 .



Generating dynamiclib

Finally, we need to use the -dynamiclib option to specify that the compiler shouldn't produce a standard executable, but should produce a library. The -o option is used to name the library with the appropriate extension: libhelloworld.jnilib. We also include the object file we just generate in previous step.

$ g++ -dynamiclib -o libhelloworld.jnilib HelloWorld.o
$ ls -lart
-rw-r--r--   1 johnnywa  staff    200 Sep 29 22:52 HelloWorld.java
-rw-r--r--   1 johnnywa  staff    377 Sep 29 22:53 HelloWorld.h
-rw-r--r--   1 johnnywa  staff    214 Sep 29 22:55 HelloWorld.cpp
-rw-r--r--   1 johnnywa  staff    442 Sep 30 11:57 HelloWorld.class
-rw-r--r--   1 johnnywa  staff   6504 Sep 30 12:47 HelloWorld.o
-rwxr-xr-x   1 johnnywa  staff  14764 Sep 30 12:51 libhelloworld.jnilib
drwxr-xr-x  14 johnnywa  staff    476 Sep 30 12:51 .

Run Java to call C++ Class



Since the java file and libhelloworld.jnilib is sitting in the same directory

$ java HelloWorld

Oh Johnny, how handsome you are!




Using Eclipse to in Linux ( Ubuntu )



C++ Shared Library Project for C++ object

Using in Eclipse is the same just create a C++ project but using "Shared Library" and Choose "Linux C++", the copy HelloWorld.h under src.





















and Create a new Class file call "HelloWorld.cpp" under src as well, then put the same content like cli in OS X.












but remember we need to reference the jni.h path ?, here you add includes via right click project and select properties. 

The reference path can be as below.

















Double check , whether you "check" - "Shared"
























Then you set up run configuration as below, especially double check your C++ Application: (Path).






















Last, one of the key is select "-fPIC", this is the key when you compiler in Linux.





















After you right click build the project, you should be able see there has new *.so shared library under debug folder.

















PS: if you got the required C++11 error, you can add compiler parameter as below.












Java Project reference C++ *.so


Then, back to your Java project, if you don't just create a java project then copy the HelloWorld.java we created before copy to the project.

But one of the key here is we would like to add *.so ( shared object ) in this project.

First of all we would like to double check what's the "Java Library Path". we can try the "System.out.println(System.getProperty("java.library.path"));"
















if you can't find the path where you local your *.so file, you can setup as below in your java environment.





Run java project in command line


If you want to run the java in command line, you might export the java project in to a Runnable jar - using Eclipse IDE. Select the main or running class file - Launch configuration. In Library handling - select the option [ Extract required libraries in to jar file ].

Open command prompt go to the directory where runnable jar is available and run 
#java -jar *.jar

BUT if you reference *.so ( C++ share objecdt ), you need to figure out the jni path as above command "System.out.println(System.getProperty("java.library.path"));", e.g.: /usr/lib. then copy your *.so to /usr/lib, otherwise your java code will be error out when you run share C++ library.

Again, this POC lab and Eclipse environment setup, I think you can complete it in 5 ~ 20 min, but be aware for the basic logic is more important to make the C++ work on Java.



Hope you enjoy it !

PS: Run Java with Error : out of memory.

If you ran into the out of memory error, which means your code might have memory leak or the java heap memory simply is not big enough for caching the object. 

For the memory leak, remember garbage collection in your code.

For java heap is not big enough, you can enlarge via the link as below.

http://www.mkyong.com/eclipse/eclipse-java-lang-outofmemoryerror-java-heap-space/

Reference:

http://mrjoelkemp.com/2012/01/getting-started-with-jni-and-c-on-osx-lion/

If you gave your best, any outcome will be perfect in its own way. - (Whiplash), 2014


Monday, September 21, 2015

How to generate tempurl for Swift object in Openstack.

tempurl is a good feature when you are using object storage, since you can generate a link and share your object to the people quick. They can download your object (file) directly from the link without credentail within the expiration time.

However, I was struggling for tempurl for pass days. Because during the days, I was dispirited and can't find too much logical info. People said about tempurl is more complicated than what's I thought and didn't explain the basic logic but just the cli (command lines).


Other than that, the some reference links includes the unit test cli but didn't give an overall summary. Here, I will explain the how curl and swift-client works to operate the tempurl and give a unit test example.


In sum, I really want give a quick intro about the basic logic, and after you aware of it, the cli for you will be logical as nature. Here are the items I would like to talk.

  1. tempurl basic logic with python example
  2. tempurl Lab Setup 
    1. Server: Swift AIO setup
    2. Client: curl / swift-client setup
  3. tempurl configuration.
  4. tempurl unit test
  5. swift over swift vs swift over ceph.
    1. swift on ceph won't work regarding the limitation

1. tempurl basic logic with python example

For generate the tempurl from OpenStack Swift, the basic idea is you have a ID ( key ) with your account then you know where is your object your would like to share with other people. After that you hash the key and a expiration date/time with the link ( url ). That's it !
  1. Key
  2. Path
  3. Expiration Date/Time

python example

The python example as below explain all the logics.












The info you need to assign.
  1. method: eg: GET
  2. host: if you test in your POC SAIO VM, you can give "http://127.0.0.1:8080"
  3. expiration: give the seconds for temp url expired.
  4. path: the path for your object, eg: '/v1/AUTH_test/testCon/test.txt'
  5. key: the key you would like to have or assign, eg: 'secret'
  6. sig: generate signature for combining 1, 2, 3 and 4
  7. tempurl: combine with hostname, object directory and temp_url_sign
--------python code start--------
'''
Created on Sep 21, 2015

@author: johnnywang
'''

import hmac
from hashlib import sha1
from time import time

#1
method = 'GET'

#2
host = "http://127.0.0.1:8080"

#3
duration_in_seconds = 6000  # Duration for which the url is valid
expires = int(time() + duration_in_seconds)

#4
path = '/v1/AUTH_test/testCon/test.txt'

#5
key = 'secret'

hmac_body = '%s\n%s\n%s' % (method, expires, path)
hmac_body = hmac.new(key, hmac_body, sha1).hexdigest()

#6
sig = hmac.new(key, hmac_body, sha1).hexdigest()

#7
rest_uri = "{host}{path}?temp_url_sig={sig}&temp_url_expires={expires}".format(
            host=host, path=path, sig=sig, expires=expires)
print rest_uri

--------python code end--------


output:

http://127.0.0.1:8080/v1/AUTH_test/testCon/test.txt?temp_url_sig=efbd2cad15e098d2327b8c7109886882f4a7afec&temp_url_expires=1442870614

2. tempurl Lab setup ( Swift over Swift )

SAIO setup - Server

SAIO is swift all in one lab, you can follow the links as below I present before, setup a POC lab for verify tempurl quick. It's quick and easy.
PS: you can configure port forwarding then you can ssh to your virtualbox SAIO vm.


There has two major client tools which can operate the swift, there are curl and swift-client.


curl setup - Client

in ubuntu

#sudo apt-get install curl libcurl3 libcurl3-dev


in mac

#brew install curl


swift-client - Client

in ubuntu

#sudo aptitude install python-pip
#sudo pip install python-swiftclient


in mac

#sudo easy_install pip
#sudo pip install --upgrade setuptools
#sudo pip install python-swiftclient

http://thornelabs.net/2014/10/29/installing-python-swiftclient-on-os-x-yosemite.html 

3. tempurl configuration

The key for make tempurl works is inject a key or second key in account level metadata. You can use curl or swift-client. Both should be all working well.


Get token and endpoint url via Swift-client


  • Get Auth-token or you username and password directly
swift@swift-VirtualBox:~$ swift -A http://127.0.0.1:8080/auth/v1.0 -U swift -K swift auth
export OS_STORAGE_URL=http://127.0.0.1:8080/v1/AUTH_swift
export OS_AUTH_TOKEN=AUTH_tk5050d2a92c43422e871d60cffa309022

swift@swift-VirtualBox:~$ swift -A http://127.0.0.1:8080/auth/v1.0 -U swift -K swift auth -v

export ST_AUTH=http://127.0.0.1:8080/auth/v1.0
export ST_USER=swift
export ST_KEY=swift

  • Once you get the endpoint and token you can insert a key into account's metadata (X-Account-Meta-Temp-Url-Key: secret)
  • Or you can access swift via username and password directly.
swift@swift-VirtualBox:~$ swift -A http://127.0.0.1:8080/auth/v1.0 -U test:tester -K testing stat
                    Account: AUTH_test
                 Containers: 2
                    Objects: 0
                      Bytes: 0
Containers in policy "gold": 2
   Objects in policy "gold": 0
     Bytes in policy "gold": 0
          Meta Temp-Url-Key: secret
                X-Timestamp: 1439949170.11303
                 X-Trans-Id: tx81e1acbb06f544af90429-0055d5654d
               Content-Type: text/plain; charset=utf-8

              Accept-Ranges: bytes


Inject tempurl key



using swift-client

post "Temp-Url-Key"
swift@swift-VirtualBox:~$ swift --os-auth-token AUTH_tk29ec321ad87b43d0bfd8a7b687ab4a2f --os-storage-url http://127.0.0.1:8080/v1/AUTH_test post -m "Temp-URL-Key: secret"

Double check

using curl with token
swift@swift-VirtualBox:~$ curl -v -H 'X-Auth-Token: AUTH_tk71106acb07784da1859cd2e434eba109' http://127.0.0.1:8080/v1/AUTH_test/
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> GET /v1/AUTH_test/ HTTP/1.1
> User-Agent: curl/7.35.0
> Host: 127.0.0.1:8080
> Accept: */*
> X-Auth-Token: AUTH_tk71106acb07784da1859cd2e434eba109
< HTTP/1.1 200 OK
< X-Account-Storage-Policy-Gold-Bytes-Used: 0
< Content-Length: 19
< X-Account-Storage-Policy-Gold-Object-Count: 0
< X-Account-Object-Count: 0
< X-Timestamp: 1439949170.11303
X-Account-Meta-Temp-Url-Key: secret
< X-Account-Storage-Policy-Gold-Container-Count: 2
< X-Account-Bytes-Used: 0
< X-Account-Container-Count: 2
< Content-Type: text/plain; charset=utf-8
< Accept-Ranges: bytes
< X-Trans-Id: tx1d08eb3202cb44a49bc6e-0055d409fd
< Date: Wed, 19 Aug 2015 04:45:49 GMT
testCon
testFolder
* Connection #0 to host 127.0.0.1 left intact

swift@swift-VirtualBox:~$ 

via token
swift@swift-VirtualBox:~$ swift --os-auth-token AUTH_tk29ec321ad87b43d0bfd8a7b687ab4a2f --os-storage-url http://127.0.0.1:8080/v1/AUTH_test stat
                    Account: AUTH_test
                 Containers: 2
                    Objects: 0
                      Bytes: 0
Containers in policy "gold": 2
   Objects in policy "gold": 0
     Bytes in policy "gold": 0
          Meta Temp-Url-Key: secret
                X-Timestamp: 1439949170.11303
                 X-Trans-Id: tx2a025cd404df4b8c9c8c4-0055d55f12
               Content-Type: text/plain; charset=utf-8
              Accept-Ranges: bytes

via username/password
swift@swift-VirtualBox:~$ swift -A http://127.0.0.1:8080/auth/v1.0 -U test:tester -K testing stat
                    Account: AUTH_test
                 Containers: 2
                    Objects: 0
                      Bytes: 0
Containers in policy "gold": 2
   Objects in policy "gold": 0
     Bytes in policy "gold": 0
          Meta Temp-Url-Key: secret
                X-Timestamp: 1439949170.11303
                 X-Trans-Id: tx81e1acbb06f544af90429-0055d5654d
               Content-Type: text/plain; charset=utf-8

              Accept-Ranges: bytes

As long as you can see the key in account level metadata, you can use the python script in above to put the path for the object and generate the tempurl.


PS: above is V1 authentication, here is V2 example if your keystone server is running V2 authentication.


swift@swift-VirtualBox:~$ swift -V 2.0 -A http://127.0.0.1:8080/auth/v2.0 -U swift -K swift post -m Temp-URL-Key:secret

swift@swift-VirtualBox:~$ swift -V 2.0 -A http://127.0.0.1:8080/auth/v2.0 -U swift -K swift stat -v

4. tempurl unit test

Here is the example you can use tempcurl to download the file, or you can copy the whole tempurl hyper-link and put on any browser to get the file.

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    15  100    15    0     0    450      0 --:--:-- --:--:-- --:--:--   483

swift@swift-VirtualBox:~$ cat download.txt 
This is a TEST

PS: since download link with hash, you might want to use curl -L to trust the hash as a part of url, and -O is output into what's the name you would like to have for the object content.

5. tempurl for Swift over Ceph 

Except the swift over swift, you can run swift over ceph which means leverage radosgateway for accessing your object in ceph. For POC, you can setup a Ceph lab follow these links.
tempurl available in RGW (http://ceph.com/docs/master/radosgw/swift/tempurl). unfortunately, it's not working in my lap or the environment I aware. 

This is what’s I realized. For temp url, we need to inject a temp-url-key in account metadata, but in Swift over ceph (http://ceph.com/docs/master/radosgw/swift/) API, it doesn’t seem to allow us to change it but only read it.  Without temp-url-key in account metadata, the tempurl will never work. 

Feature
Status
Remarks
Authentication
Supported

Get Account Metadata
Supported
No custom metadata
Swift ACLs
Supported
Supports a subset of Swift ACLs
List Containers
Supported

Delete Container
Supported

Create Container
Supported

Get Container Metadata
Supported

Update Container Metadata
Supported

Delete Container Metadata
Supported

List Objects
Supported

Static Website
Not Supported

Create Object
Supported

Create Large Object
Supported

Delete Object
Supported

Get Object
Supported

Copy Object
Supported

Get Object Metadata
Supported

Update Object Metadata
Supported

Expiring Objects
Not Supported

Object Versioning
Not Supported

CORS
Not Supported


So if you know above statement is wrong or you figure the way out, please share with me.

Reference: