Problem
I am trying to install psuedo-distributed CDH without using CDM. Everything "works" through the console. However, secondly, I start using Hue, I get an error when trying to work with Pig.
Error indicated in Hue:
JA017: Could not find running hasoop Job ID [job_local2125047777_0001], which was associated with the action [0000000-160112011607704-oozie-oozi-W @ pig]. Failure to comply with this action!
I believe this is an error that occurs due to misunderstanding due to an Oozie workflow problem connecting Pig to the job history server.
Before that, I could not use Hive from Hue, because Oozie had a hard time installing the sharelib for Oozie on HDFS. I solved this by creating a symbolic link between /etc/hadoop/conf/core-site.xml and /etc/oozie/conf/hadoop-conf/core-site.xml . As suggested here: Apache Oozie Failed to Download ShareLib
Script info
The script configuration I wrote to install CDH on Scientific Linux 7 is available here: https://github.com/coatless/stat490uiuc/blob/master/install_scripts/cdh_build.sh
In particular, I am trying to get results from a pig script:
data = LOAD '/user/hue/pig/examples/data/midsummer.txt' as (text:CHARARRAY); upper_case = FOREACH data GENERATE org.apache.pig.piggybank.evaluation.string.UPPER(text); STORE upper_case INTO '$output' ;
Attempts to solve
From googling, I came across the following solutions that were not developed after implementation.
- JA017: Could not find the running Jobo ID .
The following commands are suggested:
sudo -u hdfs hadoop fs -mkdir -p /user/history sudo -u hdfs hadoop fs -chmod -R 1777 /user/history sudo -u hdfs hadoop fs -chown mapred:hadoop /user/history
Reboot the resource and Node Manager, HDFS and history server to no avail.
Another user appeared in the stream, who suggested setting the property in job.properties , which was indicated by user.name=mapred . However, I could not find links to job.properties for Hue to work.
These messages suggest declaring fixed paths for the history server in the mapred-site.xml :
<property> <name>mapreduce.jobhistory.done-dir</name> <value>/user/history/done</value> </property> <property> <name>mapreduce.jobhistory.intermediate-done-dir</name> <value>/user/history/done_intermediate</value> </property>
This also did not work.
Indicates that the problem may be related to the permission problem, but the user does not provide details on how the problem was resolved.
Any help would be appreciated.
Complete oozie magazine
Full error text from oozie.log file:
2016-01-11 23:51:59,195 WARN ParameterVerifier:523 - SERVER[server-name] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] The application does not define formal parameters in its XML definition 2016-01-11 23:51:59,275 WARN LiteWorkflowAppService:523 - SERVER[server-name] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] libpath [hdfs://localhost:8020/user/hue/oozie/workspaces/_cloudera_-oozie-1-1452577913.73/lib] does not exist 2016-01-11 23:51:59,572 INFO ActionStartXCommand:520 - SERVER[server-name] USER[cloudera] GROUP[-] TOKEN[] APP[pig-app-hue-script] JOB[0000000-160111235108256-oozie-oozi-W] ACTION[ 0000000-160111235108256-oozie-oozi-W@ :start:] Start action [ 0000000-160111235108256-oozie-oozi-W@ :start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10] 2016-01-11 23:51:59,595 INFO ActionStartXCommand:520 - SERVER[server-name] USER[cloudera] GROUP[-] TOKEN[] APP[pig-app-hue-script] JOB[0000000-160111235108256-oozie-oozi-W] ACTION[ 0000000-160111235108256-oozie-oozi-W@ :start:] [*** 0000000-160111235108256-oozie-oozi-W@ :start:***]Action status=DONE 2016-01-11 23:51:59,596 INFO ActionStartXCommand:520 - SERVER[server-name] USER[cloudera] GROUP[-] TOKEN[] APP[pig-app-hue-script] JOB[0000000-160111235108256-oozie-oozi-W] ACTION[ 0000000-160111235108256-oozie-oozi-W@ :start:] [*** 0000000-160111235108256-oozie-oozi-W@ :start:***]Action updated in DB! 2016-01-11 23:52:00,052 INFO ActionStartXCommand:520 - SERVER[server-name] USER[cloudera] GROUP[-] TOKEN[] APP[pig-app-hue-script] JOB[0000000-160111235108256-oozie-oozi-W] ACTION[ 0000000-160111235108256-oozie-oozi-W@pig ] Start action [ 0000000-160111235108256-oozie-oozi-W@pig ] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10] 2016-01-11 23:52:03,487 WARN Credentials:96 - SERVER[server-name] Null token ignored for oozie mr token 2016-01-11 23:52:03,506 WARN Credentials:96 - SERVER[server-name] Null token ignored for oozie mr token 2016-01-11 23:52:03,562 WARN JobResourceUploader:64 - SERVER[server-name] Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2016-01-11 23:52:03,563 WARN JobResourceUploader:171 - SERVER[server-name] No job jar file set. User classes may not be found. See Job or Job