Access secure Kerberos WebHDFS without SPnego

Question

I have a working application for managing HDFS using WebHDFS. I need to do this on a Kerberos protected cluster.

The problem is that there is no library or extension to negotiate a ticket for my application, I only have a basic HTTP client.

Is it possible to create a Java service that will handle ticket exchange, and as soon as it receives a service ticket, just to transfer it to the application for use in the HTTP request? In other words, my application will ask the Java service to agree on tickets, and it will return the service ticket back to my application in a string or raw string, and the application just binds it to an HTTP request?

EDIT: Is there a similar similar solution like @SamsonScharfrichter described for HTTPfs? (As far as I know, it does not support delegation tokens)

EDIT2: Hi guys, I'm still completely lost. I am trying to find a Hadoop-auth client with no luck. Could you help me again? I read about it for hours without luck. Examples are said to do this:

* // establishing an initial connection * * URL url = new URL("http://foo:8080/bar"); * AuthenticatedURL.Token token = new AuthenticatedURL.Token(); * AuthenticatedURL aUrl = new AuthenticatedURL(); * HttpURLConnection conn = new AuthenticatedURL(url, token).openConnection(); * .... * // use the 'conn' instance * .... 

I lost already here. What initial connection do I need? how

 new AuthenticatedURL(url, token).openConnection(); 

take two parameters? for such a case there is no constructor. (this causes an error). Should someone point someone? It will probably not be so simple.

  URL url = new URL("http://<host>:14000/webhdfs/v1/?op=liststatus"); AuthenticatedURL.Token token = new AuthenticatedURL.Token(); HttpURLConnection conn = new AuthenticatedURL(url, token).openConnection(url, token); 

Answer 1

Using the Java code plus the Hadoop API to open a Kerberized session, get the delegation token for the session and pass that token to another application - as @tellisnz suggested - has a drawback: the Java API requires quite a few dependencies (i.e. a lot of JARs, and also native Hadoop libraries). If you run the application on Windows, in particular, it will be a difficult trip.

Another option is to use Java code plus WebHDFS to run one SPNEGOed request and get the delegation identifier, and then pass it to another application - this parameter requires absolutely no Hadoop library on your server , the barebones version will look like

 URL urlGetToken = new URL("http://<host>:<port>/webhdfs/v1/?op=GETDELEGATIONTOKEN") ; HttpURLConnection cnxGetToken =(HttpURLConnection) urlGetToken.openConnection() ; BufferedReader httpMessage = new BufferedReader( new InputStreamReader(cnxGetToken.getInputStream()), 1024) ; Pattern regexHasToken =Pattern.compile("urlString[\": ]+(.[^\" ]+)") ; String httpMessageLine ; while ( (httpMessageLine =httpMessage.readLine()) != null) { Matcher regexToken =regexHasToken.matcher(httpMessageLine) ; if (regexToken.find()) { System.out.println("Use that template: http://<Host>:<Port>/webhdfs/v1%AbsPath%?delegation=" +regexToken.group(1) +"&op=...") ; } } httpMessage.close() ; 

This is what I use to access HDFS from a Windows Powershell script (or even an Excel macro). Caution: with Windows, you must create your Kerberos TGT on the fly by passing the JAM JAAS configuration pointing to the corresponding keytab file. But this disclaimer also applies to the Java API.

Answer 2

You can take a look at the hadoop-auth client and create a service that performs the first connection, then you can probably grab the "Authorization" and "X-Hadoop-Delegation-Token" headers and cookies from it and add it to your main client requests .

First you will need to either use kinit to authenticate your user for the application before launching. Otherwise, you will need to make a JAAS login for your user, this tutorial gives a pretty good overview of how to do this.

Then, to log in to WebHDFS / HttpFS, we need to do something like:

 URL url = new URL("http://youhost:8080/your-kerberised-resource"); AuthenticatedURL.Token token = new AuthenticatedURL.Token(); HttpURLConnection conn = new AuthenticatedURL().openConnection(url, token); String authorizationTokenString = conn.getRequestProperty("Authorization"); String delegationToken = conn.getRequestProperty("X-Hadoop-Delegation-Token"); ... // do what you have to to get your basic client connection ... myBasicClientConnection.setRequestProperty("Authorization", authorizationTokenString); myBasicClientConnection.setRequestProperty("Cookie", "hadoop.auth=" + token.toString()); myBasicClientConnection.setRequestProperty("X-Hadoop-Delegation-Token", delegationToken);