Geek Answers Handbook

Describe an extended table in Hive

I save the table as a SequenceFile format, and I set the following commands to enable Sequence with BLOCK Compression -

set mapred.output.compress=true; set mapred.output.compression.type=BLOCK; set mapred.output.compression.codec=org.apache.hadoop.io.compress.LzoCodec;

But when I tried looking at tables like this -

 describe extended lip_table

I got information that has a field called compressed that is set to false . Thus, my data is not compressed by setting three commands:

 Detailed Table Information Table(tableName:lip_table, dbName:default, owner:uname, createTime:1343931235, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols: [FieldSchema(name:buyer_id, type:bigint, comment:null), FieldSchema(name:total_chkout, type:bigint, comment:null), FieldSchema(name:total_errpds, type:bigint, comment:null)], location:hdfs://ares-nn/apps/hdmi/uname/lip-data, inputFormat:org.apache.hadoop.mapred.SequenceFileInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat, **compressed:false**, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters: {serialization.format= , field.delim=
+4
source share
2 answers

I found this article which I think gives a solution to your problem. You should rather try to specify the use of your compression codec at the definition level of your table, either when creating the table or using the ALTER statement.

At the time of creation:

  CREATE EXTERNAL TABLE lip_table ( column1 string , column2 string ) PARTITIONED BY (date string) ROW FORMAT DELIMITED FIELDS TERMINATED BY "\t" STORED AS INPUTFORMAT "com.hadoop.mapred.DeprecatedLzoTextInputFormat" OUTPUTFORMAT "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat" LOCATION '/path/to/hive/tables/lip';

Using ALTER (affects only partitions created later):

 ALTER TABLE lip_table SET FILEFORMAT INPUTFORMAT "com.hadoop.mapred.DeprecatedLzoTextInputFormat" OUTPUTFORMAT "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat";

http://www.mrbalky.com/2011/02/24/hive-tables-partitions-and-lzo-compression/

+2
source

To avoid the serde exception, use the serde class.

 ALTER TABLE <<table name>> SET FILEFORMAT INPUTFORMAT "<<Input format class>>" OUTPUTFORMAT "<<Output format class>>" SERDE "<<Serde class>>";
+1
source

More articles:


All Articles