Geek Answers Handbook

Can I rename oozie job name dynamically

We have a Hadoop service in which we have several applications. We need to process the data for each of the applications, reworking the same workflow. They must be performed at the same time of day. The problem is that when these tasks work, it is difficult to understand for which application the work is being performed / failed / completed. Of course, I can open this work and find out it, but it takes a lot of time, since 10s of applications work under this service.

Is there any parameter in oozie to dynamically pass the name of the workflow (or part of it) during the job, for example

oozie job -run -config <filename> -name "<NameIWishToGive>" OR oozie job -run -config <filename> -nameSuffix "<MyApplicationNameUnderTheService>"

In addition, we do not want to create several task folders for execution separately, as this would be a too large copy.

Please offer.

+4
source share
4 answers

in apache docs you will find a whole bunch of oozie command lines here . I am not sure which one you are looking for, so I thought that just paste the link. hope this helps!

+1
source

It seems to me that you should just use the properties specified in the job configuration.

I managed to get a dynamic name by following these steps.

Here is an example of my workflow.xml:

 <workflow-app xmlns="uri:oozie:workflow:0.2" name="map-reduce-wf-${environment}"> ... </workflow-app>

And in my job.properties I had:

 ... environment=test ...

The name turned out to be: "map-reduce-wf-test"

+3
source

I could not find anything in this. Here is a script that finds / replaces # {appName} and # {frequency} in * .xml files + loads all files in hdf. Values ​​are taken from the properties file passed to the script as the third argument.

Gist - https://gist.github.com/epishkin/5952522

Example:

 ./upload.sh simple_reports namenode01 simple_reports/coordinator_script-1.properties

where 'simple_reports' is the folder with the files workflow.xml and Coordinator.xml.

workflow.xml:

 <workflow-app name="#{appName}" xmlns="uri:oozie:workflow:0.3"> ... </workflow-app>

coordinator.xml:

 <coordinator-app name="#{appName}-coord" xmlns="uri:oozie:coordinator:0.2" frequency="#{frequency}" start="${start}" end= "${end}" timezone="America/New_York"> ... </coordinator-app>

coordinator_script-1.properties:

 appName=multi_network frequency=${coord:days(7)} ...

Hope this helps.

0
source

I recently ran into this problem and all of these tables use the same workflow, but the oozie application name should reflect the name of the table being processed.

Then pass the same parameter from job.properties, then the ozzie application name will match dataload_tablename.

0
source

More articles:


All Articles