Java.io.IOException: invalid int encoding

Question

I am writing a mapReduce Job to read and process Avrofile. The input file is Avro Output Format - Avro

When I do the Mapreduce job, I get the following exception in the reducer phase. Since the reducer throws an IOException, I cannot catch it and reduce it in the reducer. Hue error stack trace looks like

java.io.IOException: Invalid int encoding at org.apache.avro.io.DirectBinaryDecoder.readInt(DirectBinaryDecoder.java:113) at org.apache.avro.io.ValidatingDecoder.readInt(ValidatingDecoder.java:83) at org.apache.avro.reflect.ReflectDatumReader.readInt(ReflectDatumReader.java:166) at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:156) at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:177) at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:148) at org.apache.avro.generic.GenericDatumReader.readArray(GenericDatumReader.java:206) at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:150) at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:177) at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:148) at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:139) at org.apache.avro.hadoop.io.AvroDeserializer.deserialize(AvroDeserializer. 

When searching on Google, I noticed that there is an Apache JIRA ticket ( https://issues.apache.org/jira/browse/AVRO-882 ). No updates.

I am using AVRO-1.7.5, and below is the maven dependency

  <dependency> <groupId>org.apache.avro</groupId> <artifactId>avro</artifactId> <version>1.7.5</version> </dependency> 

Any help would be greatly appreciated ?. Thanks

Answer 1

In my case, this problem manifested itself because I did not use the correct scheme to read the document.

Remember that if you are reading an Avro document that was written using a different scheme (writing scheme) than the one you are reading with, you will need to create a SpecificDatumReader using a constructor that accepts both a reading scheme and a writing scheme.

 new SpecificDatumReader<>(writerSchema, readerSchema) 

instead

 new SpecificDatumReader<>(class)