Hive / HBase integration - Zookeeper session closes immediately

Question

We have an 8 node cluster using CDH3u2 configured using Cloudera Manager. We have a dedicated node wizard that launches our only zookeeper instance. When I configure hive to start local chaos running from the master node, I have no problem returning data from HBase. When I run a distributed map / pruning through the bush, I get the following error when the slaves connect to the zookeeper.

HBase can connect to ZooKeeper, but the connection closes immediately. This may be a sign that the server has too many connections (default is 30).

We tried to set the maximum connections above (we even tried to remove the limit). This is a development cluster in which there are very few users, I know that the problem is not that there are too many connections (I can connect to zookeeper from subordinate nodes using. / ZkCli).

Server-side logs indicate that the session was terminated by the client.

The client side report on the client side says: 'Caused by: org.apache.zookeeper.KeeperException $ ConnectionLossException: KeeperErrorCode = ConnectionLoss for / hbase

Any idea why I can't connect to zookeeper via Hive Map / Reduce?

Configurations for hbase and zookeeper:

# Autogenerated by Cloudera SCM on Wed Dec 28 08:42:23 CST 2011 tickTime=2000 initLimit=10 syncLimit=5 dataDir=/var/zookeeper clientPort=2181 maxClientCnxns=1000 minSessionTimeout=4000 maxSessionTimeout=40000 

HBase Site-XML:

 <property> <name>hbase.rootdir</name> <value>hdfs://alnnimb01:8020/hbase</value> <description>The directory shared by region servers. Should be fully-qualified to include the filesystem to use. Eg: hdfs://NAMENODE_SERVER:PORT/HBASE_ROOTDIR</description> </property> <property> <name>hbase.master.port</name> <value>60000</value> <description>The port master should bind to.</description> </property> <property> <name>hbase.cluster.distributed</name> <value>true</value> <description>The mode the cluster will be in. Possible values are false: standalone and pseudo-distributed setups with managed Zookeeper true: fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh)</description> </property> <property> <name>hbase.master.info.port</name> <value>60010</value> <description>The port for the hbase master web UI Set to -1 if you do not want the info server to run.</description> </property> <property> <name>zookeeper.znode.parent</name> <value>/hbase</value> <description>Root ZNode for HBase in ZooKeeper. All of HBase ZooKeeper files that are configured with a relative path will go under this node. By default, all of HBase ZooKeeper file path are configured with a relative path, so they will all go under this directory unless changed.</description> </property> <property> <name>zookeeper.znode.rootserver</name> <value>root-region-server</value> <description>Path to ZNode holding root region location. This is written by the master and read by clients and region servers. If a relative path is given, the parent folder will be ${zookeeper.znode.parent}. By default, this means the root location is stored at /hbase/root-region-server.</description> </property> <property> <name>hbase.zookeeper.property.clientPort</name> <value>2181</value> <description>The ZooKeeper client port to which HBase clients will connect</description> </property> <property> <name>hbase.zookeeper.quorum</name> <value>alnnimb01.aln.experian.com</value> <description>Comma separated list of servers in the ZooKeeper Quorum. For example, "host1.mydomain.com,host2.mydomain.com,host3.mydomain.com".</description> 

Answer 1

It turns out that Map / Reduce, introduced by Hive, is trying to connect to zookeeper in 'localhost', regardless of how zookeeper.quorom is configured in the configuration file. I changed / etc / hosts to specify the "localhost" alias to the IP address of my node wizard and a connection to zookeeper is supported. Still looking for a better resolution, but for now it will work.

Answer 2

I get it. This was a configuration issue (as I expected). The solution was as follows:

-set 'hbase.zookeeper.quorum in the file "hive-site.xml" and put it in the directory "hadoop-conf"

What threw me away is that hive-default.xml does not have hbase .zookeeper.quorum '. I played with hive .zookeeper.quorum ', which did not match the correct configuration.

Answer 3

I apologize for the answer. I wanted to comment on the previous answer, but the comments user interface seems to be gone. > <...

In any case, I wanted to say that I have the same problem, and this is solved by hacking / etc / hosts, but this seems like a very dirty solution ...

Has anyone figured out a way to fix this purely ... ??

Thanks:)!