Geek Answers Handbook

Hbase RPC Server Timeout

I am running Hbase 1.0.1 / Hadoop 2.5.2. I am trying to run a scan on a table, but I am getting RPC timeouts.

I changed the Hbase RPC timeout to 2 minutes so I can confirm frm UI ...

<property> <name>hbase.rpc.timeout</name> <value>120000</value> <source>hbase-site.xml</source> </property>

... but my client is still disconnecting after 60 seconds ...

 Caused by: java.io.IOException: Call to xxxxxxx/172.16.5.13:16020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=2968, waitTime=60001, operationTimeout=60000 expired. at org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1235) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1203) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:216) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:300) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:31751) at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:199) at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126) ... 6 more Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=2968, waitTime=60001, operationTimeout=60000 expired. at org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:70) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1177) ... 12 more

I tried to resize the cache block, but that does not seem to make any difference.

Is there any other timeout that I am missing. There are many rows (millions) in the table, although the scan returns only 10 thousand, but the problem seems to be related only to a certain set of regions.

+5
source share
4 answers

Try this if you are creating a client-side connection and not in hbase-site.xml :

 conf.set("hbase.rpc.timeout", "1800000")
+4
source

Based on the first answers. I configured the hbase.client.scanner.timeout.period and hbase.rpc.timeout properties before opening a connection to the HBase cluster:

 import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hbase.HBaseConfiguration; Configuration conf = HBaseConfiguration.create(); conf.set("hbase.rpc.timeout", "1800000"); conf.set("hbase.client.scanner.timeout.period", "1800000");
+3
source

When the server receives an RPC scan request, it is assumed that the time limit is half the smaller of the two values: hbase.client.scanner.timeout.period and hbase.rpc.timeout (both default to 60,000 milliseconds or one minute). This is why after setting 2 minutes, the scan time is scanned after 60 seconds.

When the deadline is reached, the server returns the results accumulated to this point. This result set may be empty. If your usage pattern includes that scanning takes more than a minute, you can increase these values.

So that the waiting time is not too short, you can set hbase.cells.scanned.per.heartbeat.check to the minimum number of cells that must be scanned before the timeout check occurs. The default value is 10000. The lower the value, the more often the time is checked.

The links below may be useful for setting timeouts for an Hbase scan:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_installing_manually_book/content/best-practices-timeouts-phoenix.html

https://www.cloudera.com/documentation/enterprise/5-5-x/topics/admin_hbase_scanner_heartbeat.html#concept_xsl_dz1_jt

+1
source

That should work.

 hbase org.apache.hadoop.hbase.mapreduce.Export -Dhbase.client.scanner.timeout.period=600000 tbname /path/to/hdfs
0
source

More articles:


All Articles