Geek Answers Handbook

Perl Thrift Client for Hive?

I want to connect to a Hiveop-based Hiveop datastore using Perl. Hive allows you to connect through the Thrift interface ( http://wiki.apache.org/hadoop/Hive/HiveClient ), and for Perl there is a Thrift implementation (for example, http://metacpan.org/pod/Thrift::XS ). However, the only Thrift client I found is the Cassandra client.

Any ideas if such a client exists or how to create one? Maybe you can even connect without a clear definition?

(PS - there is also an ODBC / JDBC interface for Hive, but installing these modules is a headache and will be the last measure)

thanks!

+6
perl hive thrift
source share
3 answers

After some reading (most notably: blog.fingertap.org/?1a253760 ), I managed to create a Perl Thrift client and use it to query my server.

Steps:

  • Download, install and install Thrift: http://incubator.apache.org/thrift/download/ . Remember to install the code in lib / perl.

  • Download .thrift infrastructure files from Hive SVN under the control of your Hive installation ( http://svn.apache.org/viewvc/hive/ ). The files I used are: fb303.thrift, queryplan.thrift, hive_metastore.thrift and thrift_hive.thrift. I found them manually, but there may be more efficient ways to do this.

  • Generate Perl code with savings: thrift -r --gen perl hive_service.thrift
    Note. I needed to create a directory tree for the necessary inclusions and use the -I directive for this root of the tree. I got the required structure from the errors that left me, but then again, maybe more elegant ways to do this.

The following Perl code now works for me, written around the lines of a python example on the Hive client wiki:

 use Thrift; use Thrift::Socket; use Thrift::FramedTransport; use Thrift::BinaryProtocol; use lib <LOCATION OF GENERATED PERL CODE>; use ThriftHive; # init variables ($host, $port, $query) # my $socket = Thrift::Socket->new($host, $port); my $transport = Thrift::BufferedTransport->new($socket); my $protocol = Thrift::BinaryProtocol->new($transport); my $client = ThriftHiveClient->new($protocol); eval {$transport->open()}; #do something with Exceptions eval {$client->execute($query)}; for (my $i = 0; $i < $count; $i++) { my $row; eval {$row = $client->fetchOne()}; #use $row } $transport->close();
+8
source share

In case this is useful, I recently uploaded it to CPAN:

https://metacpan.org/module/Thrift::API::HiveClient

It is not yet fully documented and has no tests, but I'm glad to accept any patches that someone wants to send! :)

+2
source share

It may be useful to note that Thrift itself comes with the Perl library in the lib folder.

http://svn.apache.org/viewvc/thrift/trunk/lib/perl/lib/

+1
source share

More articles:


All Articles