Geek Answers Handbook

Hive combines the number of gears

I need to set the number of gears when performing a join operation in Hive. I do not want to install the same number of gears for each of my work. How can I call this an argument? Thanks in advance.

+7
source share
2 answers

At the top of the script file, put this:

set mapred.reduce.tasks=128
+14
source

The only way to do this at this time is to split your Hive script into several parts where heavy joins will fit in another script. Then for heavy connections you can specify a large number of gearboxes. To execute all of these scripts at once, you can use the hive -f option with all the scripts listed in the shell script, as shown below: #example of shell script RunMyHQL.sh $HIVE_HOME/bin/hive -f /somepath/script1.hql $HIVE_HOME/bin/hive -f /somepath/script2.hql $HIVE_HOME/bin/hive -f /somepath/script3.hql

Then assign the appropriate permissions to the file RunMyHQL.sh chmod u+x /somepath/RunMyHQL.sh

Then run the shell script ./RunMyHQL.sh Each of scenarios1, script2 and script3 can contain the required number of maps and reducers. If you need to do this at regular intervals over and over again, I suggest you implement the Oozie workflow.

0
source

More articles:


All Articles