I am using EMR-4.3.0, Spark 1.6.0, Hive 1.0.0.
I am writing a table like this (pseudocode) -
val df = <a dataframe>
df.registerTempTable("temptable")
sqlContext.setConf("hive.exec.dynamic.partition.mode","true")
sqlContext.sql("create external table exttable ( some columns ... )
partitioned by (partkey int) stored as parquet location 's3://some.bucket'")
sqlContext.sql("insert overwrite exttable partition(partkey) select columns from
temptable")
The write is working fine and I can read the table with -
sqlContext.sql("select * from exttable")
However, when I try to read a table using Hive as -
hive -e 'select * from exttable'
Hive throws a NullPointerException with the next stack trace. Any help appreciated! -
questTime = [0.008], ResponseProcessingTime = [0.295], HttpClientSendRequestTime = [0.313],
2016-05-19 03: 08: 02,537 ERROR [main()]: CliDriver (SessionState.java:printError(833)) - java.io.IOException: java.lang.NullPointerException
java.io.IOException: java.lang.NullPointerException org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:663) at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:561) at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:138) at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1619) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:221) org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:153) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:364) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:712) org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:631) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:570) at sun.reflect.NativeMethodAccessorImpl.invoke0 ( ) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:606) org.apache.hadoop.util.RunJar.run(RunJar.java:221) org.apache.hadoop.util.RunJar.main(RunJar.java:136)
: java.lang.NullPointerException parquet.format.converter.ParquetMetadataConverter.fromParquetStatistics(ParquetMetadataConverter.java:247) parquet.format.converter.ParquetMetadataConverter.fromParquetMetadata(ParquetMetadataConverter.java:368) parquet.format.converter.ParquetMetadataConverter.readParquetMetadata(ParquetMetadataConverter.java:346) .hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:296) .hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:254) at org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.getSplit(ParquetRecordReaderWrapper.java:200) org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper. (ParquetRecordReaderWrapper.java:79) org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper. (ParquetRecordReaderWrapper.java:66) at org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:72) at org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:498) at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:588) ... 15
. , . ?