Oozie and Job History server configuration issues

Problem

I am trying to install psuedo-distributed CDH without using CDM. Everything "works" through the console. However, secondly, I start using Hue, I get an error when trying to work with Pig.

Error indicated in Hue:

JA017: Could not find running hasoop Job ID [job_local2125047777_0001], which was associated with the action [0000000-160112011607704-oozie-oozi-W @ pig]. Failure to comply with this action!

I believe this is an error that occurs due to misunderstanding due to an Oozie workflow problem connecting Pig to the job history server.

Before that, I could not use Hive from Hue, because Oozie had a hard time installing the sharelib for Oozie on HDFS. I solved this by creating a symbolic link between /etc/hadoop/conf/core-site.xml and /etc/oozie/conf/hadoop-conf/core-site.xml . As suggested here: Apache Oozie Failed to Download ShareLib

Script info

The script configuration I wrote to install CDH on Scientific Linux 7 is available here: https://github.com/coatless/stat490uiuc/blob/master/install_scripts/cdh_build.sh

In particular, I am trying to get results from a pig script:

 data = LOAD '/user/hue/pig/examples/data/midsummer.txt' as (text:CHARARRAY); upper_case = FOREACH data GENERATE org.apache.pig.piggybank.evaluation.string.UPPER(text); STORE upper_case INTO '$output' ; 

Attempts to solve

From googling, I came across the following solutions that were not developed after implementation.

  • JA017: Could not find the running Jobo ID .

The following commands are suggested:

 sudo -u hdfs hadoop fs -mkdir -p /user/history sudo -u hdfs hadoop fs -chmod -R 1777 /user/history sudo -u hdfs hadoop fs -chown mapred:hadoop /user/history 

Reboot the resource and Node Manager, HDFS and history server to no avail.

Another user appeared in the stream, who suggested setting the property in job.properties , which was indicated by user.name=mapred . However, I could not find links to job.properties for Hue to work.

These messages suggest declaring fixed paths for the history server in the mapred-site.xml :

 <property> <name>mapreduce.jobhistory.done-dir</name> <value>/user/history/done</value> </property> <property> <name>mapreduce.jobhistory.intermediate-done-dir</name> <value>/user/history/done_intermediate</value> </property> 

This also did not work.

Indicates that the problem may be related to the permission problem, but the user does not provide details on how the problem was resolved.

Any help would be appreciated.

Complete oozie magazine

Full error text from oozie.log file:

 2016-01-11 23:51:59,195 WARN ParameterVerifier:523 - SERVER[server-name] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] The application does not define formal parameters in its XML definition 2016-01-11 23:51:59,275 WARN LiteWorkflowAppService:523 - SERVER[server-name] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] libpath [hdfs://localhost:8020/user/hue/oozie/workspaces/_cloudera_-oozie-1-1452577913.73/lib] does not exist 2016-01-11 23:51:59,572 INFO ActionStartXCommand:520 - SERVER[server-name] USER[cloudera] GROUP[-] TOKEN[] APP[pig-app-hue-script] JOB[0000000-160111235108256-oozie-oozi-W] ACTION[ 0000000-160111235108256-oozie-oozi-W@ :start:] Start action [ 0000000-160111235108256-oozie-oozi-W@ :start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10] 2016-01-11 23:51:59,595 INFO ActionStartXCommand:520 - SERVER[server-name] USER[cloudera] GROUP[-] TOKEN[] APP[pig-app-hue-script] JOB[0000000-160111235108256-oozie-oozi-W] ACTION[ 0000000-160111235108256-oozie-oozi-W@ :start:] [*** 0000000-160111235108256-oozie-oozi-W@ :start:***]Action status=DONE 2016-01-11 23:51:59,596 INFO ActionStartXCommand:520 - SERVER[server-name] USER[cloudera] GROUP[-] TOKEN[] APP[pig-app-hue-script] JOB[0000000-160111235108256-oozie-oozi-W] ACTION[ 0000000-160111235108256-oozie-oozi-W@ :start:] [*** 0000000-160111235108256-oozie-oozi-W@ :start:***]Action updated in DB! 2016-01-11 23:52:00,052 INFO ActionStartXCommand:520 - SERVER[server-name] USER[cloudera] GROUP[-] TOKEN[] APP[pig-app-hue-script] JOB[0000000-160111235108256-oozie-oozi-W] ACTION[ 0000000-160111235108256-oozie-oozi-W@pig ] Start action [ 0000000-160111235108256-oozie-oozi-W@pig ] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10] 2016-01-11 23:52:03,487 WARN Credentials:96 - SERVER[server-name] Null token ignored for oozie mr token 2016-01-11 23:52:03,506 WARN Credentials:96 - SERVER[server-name] Null token ignored for oozie mr token 2016-01-11 23:52:03,562 WARN JobResourceUploader:64 - SERVER[server-name] Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2016-01-11 23:52:03,563 WARN JobResourceUploader:171 - SERVER[server-name] No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2016-01-11 23:52:04,169 WARN MRApps:582 - SERVER[server-name] cache file (mapreduce.job.cache.files) hdfs://localhost:8020/user/oozie/share/lib/lib_20160111222734/pig/json-simple-1.1.jar conflicts with cache file (mapreduce.job.cache.files) hdfs://localhost:8020/user/oozie/share/lib/lib_20160111222734/oozie/json-simple-1.1.jar This will be an error in Hadoop 2.0 2016-01-11 23:52:08,611 WARN Credentials:96 - SERVER[server-name] Null token ignored for oozie mr token 2016-01-11 23:52:08,618 WARN PigActionExecutor:523 - SERVER[server-name] USER[cloudera] GROUP[-] TOKEN[] APP[pig-app-hue-script] JOB[0000000-160111235108256-oozie-oozi-W] ACTION[ 0000000-160111235108256-oozie-oozi-W@pig ] Exception in check(). Message[JA017: Could not lookup launched hadoop Job ID [job_local1961106749_0001] which was associated with action [ 0000000-160111235108256-oozie-oozi-W@pig ]. Failing this action!] org.apache.oozie.action.ActionExecutorException: JA017: Could not lookup launched hadoop Job ID [job_local1961106749_0001] which was associated with action [ 0000000-160111235108256-oozie-oozi-W@pig ]. Failing this action! at org.apache.oozie.action.hadoop.JavaActionExecutor.check(JavaActionExecutor.java:1274) at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1203) at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:250) at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:64) at org.apache.oozie.command.XCommand.call(XCommand.java:286) at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:321) at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:250) at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2016-01-11 23:52:08,620 WARN ActionStartXCommand:523 - SERVER[server-name] USER[cloudera] GROUP[-] TOKEN[] APP[pig-app-hue-script] JOB[0000000-160111235108256-oozie-oozi-W] ACTION[ 0000000-160111235108256-oozie-oozi-W@pig ] Error starting action [pig]. ErrorType [FAILED], ErrorCode [JA017], Message [JA017: Could not lookup launched hadoop Job ID [job_local1961106749_0001] which was associated with action [ 0000000-160111235108256-oozie-oozi-W@pig ]. Failing this action!] org.apache.oozie.action.ActionExecutorException: JA017: Could not lookup launched hadoop Job ID [job_local1961106749_0001] which was associated with action [ 0000000-160111235108256-oozie-oozi-W@pig ]. Failing this action! at org.apache.oozie.action.hadoop.JavaActionExecutor.check(JavaActionExecutor.java:1274) at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1203) at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:250) at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:64) at org.apache.oozie.command.XCommand.call(XCommand.java:286) at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:321) at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:250) at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2016-01-11 23:52:08,621 WARN ActionStartXCommand:523 - SERVER[server-name] USER[cloudera] GROUP[-] TOKEN[] APP[pig-app-hue-script] JOB[0000000-160111235108256-oozie-oozi-W] ACTION[ 0000000-160111235108256-oozie-oozi-W@pig ] Failing Job due to failed action [pig] 2016-01-11 23:52:08,623 WARN LiteWorkflowInstance:523 - SERVER[server-name] USER[cloudera] GROUP[-] TOKEN[] APP[pig-app-hue-script] JOB[0000000-160111235108256-oozie-oozi-W] ACTION[ 0000000-160111235108256-oozie-oozi-W@pig ] Workflow Failed. Failing node [pig] 2016-01-11 23:52:08,768 INFO KillXCommand:520 - SERVER[server-name] USER[cloudera] GROUP[-] TOKEN[] APP[pig-app-hue-script] JOB[0000000-160111235108256-oozie-oozi-W] ACTION[] STARTED WorkflowKillXCommand for jobId=0000000-160111235108256-oozie-oozi-W 2016-01-11 23:52:08,806 INFO KillXCommand:520 - SERVER[server-name] USER[cloudera] GROUP[-] TOKEN[] APP[pig-app-hue-script] JOB[0000000-160111235108256-oozie-oozi-W] ACTION[] ENDED WorkflowKillXCommand for jobId=0000000-160111235108256-oozie-oozi-W 2016-01-11 23:52:09,038 INFO CallbackServlet:520 - SERVER[server-name] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000000-160111235108256-oozie-oozi-W] ACTION[ 0000000-160111235108256-oozie-oozi-W@pig ] callback for action [ 0000000-160111235108256-oozie-oozi-W@pig ] 2016-01-11 23:52:09,072 ERROR CompletedActionXCommand:517 - SERVER[server-name] USER[-] GROUP[-] TOKEN[] APP[-] JOB[0000000-160111235108256-oozie-oozi-W] ACTION[ 0000000-160111235108256-oozie-oozi-W@pig ] XException, org.apache.oozie.command.CommandException: E0800: Action it is not running its in [FAILED] state, action [ 0000000-160111235108256-oozie-oozi-W@pig ] at org.apache.oozie.command.wf.CompletedActionXCommand.eagerVerifyPrecondition(CompletedActionXCommand.java:92) at org.apache.oozie.command.XCommand.call(XCommand.java:257) at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2016-01-11 23:52:09,082 WARN CallableQueueService$CallableWrapper:523 - SERVER[server-name] USER[-] GROUP[-] TOKEN[] APP[-] JOB[0000000-160111235108256-oozie-oozi-W] ACTION[ 0000000-160111235108256-oozie-oozi-W@pig ] exception callable [callback], E0800: Action it is not running its in [FAILED] state, action [ 0000000-160111235108256-oozie-oozi-W@pig ] org.apache.oozie.command.CommandException: E0800: Action it is not running its in [FAILED] state, action [ 0000000-160111235108256-oozie-oozi-W@pig ] at org.apache.oozie.command.wf.CompletedActionXCommand.eagerVerifyPrecondition(CompletedActionXCommand.java:92) at org.apache.oozie.command.XCommand.call(XCommand.java:257) at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 
+7
hadoop yarn hadoop2 cloudera-cdh hue
source share
1 answer

You should double check with the HUE File browser whether all permissions are allowed in all directories and subdirectories / user / history.

In my case, all users had permissions for all / user / history subfolders, but the HUE File browser told me that the "/ user / history" directory itself has the following permission set:

 Name User Group Permissions history mapred hadoop drwxrwx--- 

This led to an error when using a different user, and not for display. The following command helped:

 sudo -u hdfs hadoop fs -chmod 777 /user/history 
0
source share

All Articles