Job cuts

Question

I have five cards that I run separately. I want to bring them all together. Thus, the output of one task proceeds to the next task. I have currently written a shell script to execute them all. Is there any way to write this in java? Give an example.

thanks

Answer 1

You can find JobControl as the easiest way to combine these jobs together. For more complex workflows, I recommend checking out Oozie .

Answer 2

Hi I had a similar requirement One way to do this is

after sending the first job after

Job job1 = new Job( getConf() ); job.waitForCompletion( true ); 

and then check the status with

 if(job.isSuccessful()){ //start another job with different Mapper. //change config Job job2 = new Job( getConf() ); } 

Answer 3

Oozi is the solution for you. You can send types of type abbreviations, jobs for the hive, jobs for pigs, system commands, etc. Through Oozie action tags.

It even has a coordinator that acts as a cron for your workflow.

Answer 4

Another option is Cascading , which also provides an abstraction layer on top of Hadoop: itseems provides a similar combination of work-with-Hadoop-concepts, but allowing-hadoop-do-the-M / R-heavy lift, which get the use of Oozie workflows that invoke Pig scripts.