Geek Answers Handbook

Oozie - Output exceeds limit [2048]

I am trying to start a simple workflow with a hive script. This bush script just calls the join (the tables are very large ); After completing the hive script, I expected the status of the workflow to change from RUNNING to successful, but this does not happen.

This is the contents of the workflow log:

2016-05-31 15:52:34,590 WARN 

org.apache.oozie.action.hadoop.HiveActionExecutor: 
SERVER[hadoop02] U
SER[scapp] 
GROUP[-] 
TOKEN[] 
APP[wf-sqoop-hive-agreement] 
JOB[0000001-160531143657136-oozie-oozi-W] 
ACTION[0000001-160531143657136-oozie-oozi-W@hive-query-agreement] Launcher 
ERROR, reason: Main class [org.apache.oozie.action.hadoop.HiveMain], exception invoking main(), Output data exceeds its limit [2048] 2016-05-31 15:52:34,591 

WARN org.apache.oozie.action.hadoop.HiveActionExecutor: 
SERVER[hadoop02] 
USER[scapp] 
GROUP[-] 
TOKEN[] 
APP[wf-sqoop-hive-agreement] 
JOB[0000001-160531143657136-oozie-oozi-W] 
ACTION[0000001-160531143657136-oozie-oozi-W@hive-query-agreement] 
Launcher exception: Output data exceeds its limit [2048] 
org.apache.oozie.action.hadoop.LauncherException: Output data exceeds its limit [2048]  
at org.apache.oozie.action.hadoop.LauncherMapper.getLocalFileContentStr(LauncherMapper.java:415)    
at org.apache.oozie.action.hadoop.LauncherMapper.handleActionData(LauncherMapper.java:391)  
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:275) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)  
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)  
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)   
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)     
at java.security.AccessController.doPrivileged(Native Method)   
at javax.security.auth.Subject.doAs(Subject.java:415)   
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)     
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
+4
source share
4 answers

@BorderStark I do not think that a property means its size in MB. The size is in "characters", i.e. Bytes, according to the following entry in the oozie-default.xml file.

<property>
     <name>oozie.action.max.output.data</name>
     <value>2048</value>
     <description>
         Max size in characters for output data.
     </description>
 </property>
+4
source

, ?

oozie-default.xml

<property>
<name>oozie.action.max.output.data</name>
<value>XXXX</value>
</property>

Ambari:    Oozie → oozie.action.max.output.data = 4096

, , . 2048B, .

+2

I assume that you could include an element <capture-output>in the action of your bush or any other workflow actions. Try removing this item from the workflow and starting it again.

<capture-output> will hold STDOUT ssh commands and limited to 2KiB [2048 bytes]

You can learn more about it here.

+1
source

More articles:


All Articles