Hdfs - ls: Local exception error: com.google.protobuf.InvalidProtocolBufferException:

Question

I am trying to use the below to list my dirs in hdfs ...

ubuntu@ubuntu :~$ hadoop fs -ls hdfs://127.0.0.1:50075/ ls: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: "ubuntu/127.0.0.1"; destination host is: "ubuntu":50075; 

Here is my / etc / hosts file

 127.0.0.1 ubuntu localhost #127.0.1.1 ubuntu # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters 

Hi, am I using hdfs: // correctly to list my dirs?

I am using canera 4.3 on ubuntu 12.04

Answer 1

HDFS does not work on 50075. To check your hdfs port, use the following command in linux

 hdfs getconf -confKey fs.default.name 

You will get a result similar to

 hdfs://hmaster:54310 

And correct your URL accordingly

Answer 2

In your cloudera manager, check the node name for the configuration element "Name Service RPC Service Port" OR "dfs.namenode.servicerpc-address". Add the same port number there. And it should work fine.

Answer 3

Does your NN work on port 50075 ? In fact, you do not need to do this if you just want to list all the directories. Just use hadoop fs -ls / . This will list all your directories in your root directory.

Answer 4

This error occurs due to:

• It cannot contact namenode
• Namenode may not work (you can verify it by running the jps command.)
• kill what ever runs on that particular port

check what works in a particular port netstat -tulpn | grep 8080 netstat -tulpn | grep 8080 and kill -9 <PID>

4. Restart namenode

Answer 5

In / usr / local / hadoop / etc / hadoop / core-site.xml

Instead of localhost use 0.0.0.0 ie.

Change <value>hdfs://localhost:50075</value> to

 <value>hdfs://0.0.0.0:50075</value> 

This solved the problem for me

Answer 6

Can you verify your hostname ?. The same name (ubuntu) should be in the file / etc / hostname and / etc / hosts.

Answer 7

Make sure your tcp port NN is at 50075, which is defined in hdfs-site.xml

 <property> <name>dfs.namenode.rpc-address.nn1</name> <value>r101072036.sqa.zmf:9000</value> </property> 

My problem is that I use the http-address port to connect to NN, it throws the same exception as you.

The http port is also configured in hdfs-site.xml:

 <property> <name>dfs.namenode.http-address.nn1</name> <value>r101072036.sqa.zmf:8000</value> </property>