HBase cascaded output

Question

I am trying to write Scalding jobs that should connect to HBase, but I have problems using the HBase crane. I tried using the tap provided by Twitter Maple after this sample project , but it seems that there is some incompatibility between the version of Hadoop / HBase that I use and the one that was used as the Twitter client.

My cluster runs Cloudera CDH4 with HBase 0.92 and Hadoop 2.0.0-cdh4.1.3. When I run a Scalding job connecting to HBase, I get an exception

java.lang.NoSuchMethodError: org.apache.hadoop.net.NetUtils.getInputStream(Ljava/net/Socket;)Ljava/io/InputStream; at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:363) at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1046) ... 

It seems that the HBase client used by Twitter Maple is expecting some kind of method on NetUtils that does not exist in the version of Hadoop deployed in my cluster.

How to determine what exactly is a mismatch - what version would the HBase client expect, etc.? Is there any way to mitigate these problems?

It seems to me that often the client libraries are compiled with a hard-coded version of the Hadoop dependencies, and it's hard to match them with the actual versions.

Answer 1

The method actually exists, but changed its signature. It basically comes down to the fact that your client and server have different versions of the Hadoop libraries. If Cloudera is running on your server, you must use the HBase and Hadoop libraries from Cloudera. If you are using Maven, you can use the Cloudera Maven repository .

It looks like the dependencies in the library are being processed in Build.scala. I haven't used Scala yet, so I'm not quite sure how to fix it.

A change that violated compatibility was committed as part of the HADOOP-8350 . Take a look at Ted Jung's comments and answers. He is working on HBase and has the same problem. Later versions of HBase libraries should automatically handle this problem according to its comment.